howard.objects.variants API documentation

Variants( conn=None, input: str = None, output: str = None, config: dict = {}, param: dict = {}, load: bool = False) View Source

36    def __init__(
37        self,
38        conn=None,
39        input: str = None,
40        output: str = None,
41        config: dict = {},
42        param: dict = {},
43        load: bool = False,
44    ) -> None:
45        """
46        The function `__init__` initializes the variables, sets the input, output, config, param, connexion and
47        header
48
49        :param conn: the connection to the database
50        :param input: the input file
51        :param output: the output file
52        :param config: a dictionary containing the configuration of the model
53        :param param: a dictionary containing the parameters of the model
54        """
55
56        # Init variables
57        self.init_variables()
58
59        # Input
60        self.set_input(input)
61
62        # Config
63        self.set_config(config)
64
65        # Param
66        self.set_param(param)
67
68        # Output
69        self.set_output(output)
70
71        # connexion
72        self.set_connexion(conn)
73
74        # Header
75        self.set_header()
76
77        # Load data
78        if load:
79            self.load_data()

The function __init__ initializes the variables, sets the input, output, config, param, connexion and header

Parameters

conn: the connection to the database
input: the input file
output: the output file
config: a dictionary containing the configuration of the model
param: a dictionary containing the parameters of the model

def set_input(self, input: str = None) -> None: View Source

 81    def set_input(self, input: str = None) -> None:
 82        """
 83        The function `set_input` takes a file name as input, extracts the name and extension, and sets
 84        attributes in the class accordingly.
 85
 86        :param input: The `set_input` method in the provided code snippet is used to set attributes
 87        related to the input file. Here's a breakdown of the parameters and their usage in the method:
 88        :type input: str
 89        """
 90
 91        if input and not isinstance(input, str):
 92            try:
 93                self.input = input.name
 94            except:
 95                log.error(f"Input file '{input} in bad format")
 96                raise ValueError(f"Input file '{input} in bad format")
 97        else:
 98            self.input = input
 99
100        # Input format
101        if input:
102            input_name, input_extension = os.path.splitext(self.input)
103            self.input_name = input_name
104            self.input_extension = input_extension
105            self.input_format = self.input_extension.replace(".", "")

The function set_input takes a file name as input, extracts the name and extension, and sets attributes in the class accordingly.

Parameters

input: The set_input method in the provided code snippet is used to set attributes related to the input file. Here's a breakdown of the parameters and their usage in the method:

def set_config(self, config: dict) -> None: View Source

107    def set_config(self, config: dict) -> None:
108        """
109        The set_config function takes a config object and assigns it as the configuration object for the
110        class.
111
112        :param config: The `config` parameter in the `set_config` function is a dictionary object that
113        contains configuration settings for the class. When you call the `set_config` function with a
114        dictionary object as the argument, it will set that dictionary as the configuration object for
115        the class
116        :type config: dict
117        """
118
119        self.config = config

The set_config function takes a config object and assigns it as the configuration object for the class.

Parameters

config: The config parameter in the set_config function is a dictionary object that contains configuration settings for the class. When you call the set_config function with a dictionary object as the argument, it will set that dictionary as the configuration object for the class

def set_param(self, param: dict) -> None: View Source

121    def set_param(self, param: dict) -> None:
122        """
123        This function sets a parameter object for the class based on the input dictionary.
124
125        :param param: The `set_param` method you provided takes a dictionary object as input and sets it
126        as the `param` attribute of the class instance
127        :type param: dict
128        """
129
130        self.param = param

This function sets a parameter object for the class based on the input dictionary.

Parameters

param: The set_param method you provided takes a dictionary object as input and sets it as the param attribute of the class instance

def init_variables(self) -> None: View Source

132    def init_variables(self) -> None:
133        """
134        This function initializes the variables that will be used in the rest of the class
135        """
136
137        self.prefix = "howard"
138        self.table_variants = "variants"
139        self.dataframe = None
140
141        self.comparison_map = {
142            "gt": ">",
143            "gte": ">=",
144            "lt": "<",
145            "lte": "<=",
146            "equals": "=",
147            "contains": "SIMILAR TO",
148        }
149
150        self.code_type_map = {"Integer": 0, "String": 1, "Float": 2, "Flag": 3}
151
152        self.code_type_map_to_sql = {
153            "Integer": "INTEGER",
154            "String": "VARCHAR",
155            "Float": "FLOAT",
156            "Flag": "VARCHAR",
157        }
158
159        self.index_additionnal_fields = []

This function initializes the variables that will be used in the rest of the class

def get_indexing(self) -> bool: View Source

161    def get_indexing(self) -> bool:
162        """
163        It returns the value of the key "indexing" in the dictionary. If the key is not present, it
164        returns False.
165        :return: The value of the indexing parameter.
166        """
167
168        return self.get_param().get("indexing", False)

It returns the value of the key "indexing" in the dictionary. If the key is not present, it returns False.

Returns

The value of the indexing parameter.

def get_connexion_config(self) -> dict: View Source

170    def get_connexion_config(self) -> dict:
171        """
172        The function `get_connexion_config` returns a dictionary containing the configuration for a
173        connection, including the number of threads and memory limit.
174        :return: a dictionary containing the configuration for the Connexion library.
175        """
176
177        # config
178        config = self.get_config()
179
180        # Connexion config
181        connexion_config = {}
182        threads = self.get_threads()
183
184        # Threads
185        if threads:
186            connexion_config["threads"] = threads
187
188        # Memory
189        # if config.get("memory", None):
190        #     connexion_config["memory_limit"] = config.get("memory")
191        if self.get_memory():
192            connexion_config["memory_limit"] = self.get_memory()
193
194        # Temporary directory
195        if config.get("tmp", None):
196            connexion_config["temp_directory"] = config.get("tmp")
197
198        # Access
199        if config.get("access", None):
200            access = config.get("access")
201            if access in ["RO"]:
202                access = "READ_ONLY"
203            elif access in ["RW"]:
204                access = "READ_WRITE"
205            connexion_db = self.get_connexion_db()
206            if connexion_db in ":memory:":
207                access = "READ_WRITE"
208            connexion_config["access_mode"] = access
209
210        return connexion_config

The function get_connexion_config returns a dictionary containing the configuration for a connection, including the number of threads and memory limit.

Returns

a dictionary containing the configuration for the Connexion library.

def get_duckdb_settings(self) -> dict: View Source

212    def get_duckdb_settings(self) -> dict:
213        """
214        The function `get_duckdb_settings` retrieves DuckDB settings from a configuration file or a
215        string.
216        :return: The function `get_duckdb_settings` returns a dictionary object `duckdb_settings_dict`.
217        """
218
219        # config
220        config = self.get_config()
221
222        # duckdb settings
223        duckdb_settings_dict = {}
224        if config.get("duckdb_settings", None):
225            duckdb_settings = config.get("duckdb_settings")
226            duckdb_settings = full_path(duckdb_settings)
227            # duckdb setting is a file
228            if os.path.exists(duckdb_settings):
229                with open(duckdb_settings) as json_file:
230                    duckdb_settings_dict = yaml.safe_load(json_file)
231            # duckdb settings is a string
232            else:
233                duckdb_settings_dict = json.loads(duckdb_settings)
234
235        return duckdb_settings_dict

The function get_duckdb_settings retrieves DuckDB settings from a configuration file or a string.

Returns

The function get_duckdb_settings returns a dictionary object duckdb_settings_dict.

def set_connexion_db(self) -> str: View Source

237    def set_connexion_db(self) -> str:
238        """
239        The function `set_connexion_db` returns the appropriate database connection string based on the
240        input format and connection type.
241        :return: the value of the variable `connexion_db`.
242        """
243
244        # Default connexion db
245        default_connexion_db = ":memory:"
246
247        # Find connexion db
248        if self.get_input_format() in ["db", "duckdb"]:
249            connexion_db = self.get_input()
250        elif self.get_connexion_type() in ["memory", default_connexion_db, None]:
251            connexion_db = default_connexion_db
252        elif self.get_connexion_type() in ["tmpfile"]:
253            tmp_name = tempfile.mkdtemp(
254                prefix=self.get_prefix(), dir=self.get_tmp_dir(), suffix=".db"
255            )
256            connexion_db = f"{tmp_name}/tmp.db"
257        elif self.get_connexion_type() != "":
258            connexion_db = self.get_connexion_type()
259        else:
260            connexion_db = default_connexion_db
261
262        # Set connexion db
263        self.connexion_db = connexion_db
264
265        return connexion_db

The function set_connexion_db returns the appropriate database connection string based on the input format and connection type.

Returns

the value of the variable connexion_db.

def set_connexion(self, conn) -> None: View Source

267    def set_connexion(self, conn) -> None:
268        """
269        The function `set_connexion` creates a connection to a database, with options for different
270        database formats and settings.
271
272        :param conn: The `conn` parameter in the `set_connexion` method is the connection to the
273        database. If a connection is not provided, a new connection to an in-memory database is created.
274        The method then proceeds to set up the connection based on the specified format (e.g., duckdb or
275        sqlite
276        """
277
278        # Connexion db
279        connexion_db = self.set_connexion_db()
280
281        # Connexion config
282        connexion_config = self.get_connexion_config()
283
284        # Connexion format
285        connexion_format = self.get_config().get("connexion_format", "duckdb")
286        # Set connexion format
287        self.connexion_format = connexion_format
288
289        # Connexion
290        if not conn:
291            if connexion_format in ["duckdb"]:
292                conn = duckdb.connect(connexion_db, config=connexion_config)
293                # duckDB settings
294                duckdb_settings = self.get_duckdb_settings()
295                if duckdb_settings:
296                    for setting in duckdb_settings:
297                        setting_value = duckdb_settings.get(setting)
298                        if isinstance(setting_value, str):
299                            setting_value = f"'{setting_value}'"
300                        conn.execute(f"PRAGMA {setting}={setting_value};")
301            elif connexion_format in ["sqlite"]:
302                conn = sqlite3.connect(connexion_db)
303
304        # Set connexion
305        self.conn = conn
306
307        # Log
308        log.debug(f"connexion_format: {connexion_format}")
309        log.debug(f"connexion_db: {connexion_db}")
310        log.debug(f"connexion config: {connexion_config}")
311        log.debug(f"connexion duckdb settings: {self.get_duckdb_settings()}")

The function set_connexion creates a connection to a database, with options for different database formats and settings.

Parameters

conn: The conn parameter in the set_connexion method is the connection to the database. If a connection is not provided, a new connection to an in-memory database is created. The method then proceeds to set up the connection based on the specified format (e.g., duckdb or sqlite

def set_output(self, output: str = None) -> None: View Source

313    def set_output(self, output: str = None) -> None:
314        """
315        The `set_output` function in Python sets the output file based on the input or a specified key
316        in the config file, extracting the output name, extension, and format.
317
318        :param output: The `output` parameter in the `set_output` method is used to specify the name of
319        the output file. If the config file has an 'output' key, the method sets the output to the value
320        of that key. If no output is provided, it sets the output to `None`
321        :type output: str
322        """
323
324        if output and not isinstance(output, str):
325            self.output = output.name
326        else:
327            self.output = output
328
329        # Output format
330        if self.output:
331            output_name, output_extension = os.path.splitext(self.output)
332            self.output_name = output_name
333            self.output_extension = output_extension
334            self.output_format = self.output_extension.replace(".", "")
335        else:
336            self.output_name = None
337            self.output_extension = None
338            self.output_format = None

The set_output function in Python sets the output file based on the input or a specified key in the config file, extracting the output name, extension, and format.

Parameters

output: The output parameter in the set_output method is used to specify the name of the output file. If the config file has an 'output' key, the method sets the output to the value of that key. If no output is provided, it sets the output to None

def set_header(self) -> None: View Source

340    def set_header(self) -> None:
341        """
342        It reads the header of a VCF file and stores it as a list of strings and as a VCF object
343        """
344
345        input_file = self.get_input()
346        default_header_list = [
347            "##fileformat=VCFv4.2",
348            "#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO",
349        ]
350
351        # Full path
352        input_file = full_path(input_file)
353
354        if input_file:
355
356            input_format = self.get_input_format()
357            input_compressed = self.get_input_compressed()
358            config = self.get_config()
359            header_list = default_header_list
360            if input_format in [
361                "vcf",
362                "hdr",
363                "tsv",
364                "csv",
365                "psv",
366                "parquet",
367                "db",
368                "duckdb",
369            ]:
370                # header provided in param
371                if config.get("header_file", None):
372                    with open(config.get("header_file"), "rt") as f:
373                        header_list = self.read_vcf_header(f)
374                # within a vcf file format (header within input file itsself)
375                elif input_format in ["vcf", "hdr"] and not os.path.isdir(input_file):
376                    # within a compressed vcf file format (.vcf.gz)
377                    if input_compressed:
378                        with bgzf.open(input_file, "rt") as f:
379                            header_list = self.read_vcf_header(f)
380                    # within an uncompressed vcf file format (.vcf)
381                    else:
382                        with open(input_file, "rt") as f:
383                            header_list = self.read_vcf_header(f)
384                # header provided in default external file .hdr
385                elif os.path.exists((input_file + ".hdr")):
386                    with open(input_file + ".hdr", "rt") as f:
387                        header_list = self.read_vcf_header(f)
388                else:
389                    try:  # Try to get header info fields and file columns
390
391                        with tempfile.TemporaryDirectory() as tmpdir:
392
393                            # Create database
394                            db_for_header = Database(database=input_file)
395
396                            # Get header columns for infos fields
397                            db_header_from_columns = (
398                                db_for_header.get_header_from_columns()
399                            )
400
401                            # Get real columns in the file
402                            db_header_columns = db_for_header.get_columns()
403
404                            # Write header file
405                            header_file_tmp = os.path.join(tmpdir, "header")
406                            f = open(header_file_tmp, "w")
407                            vcf.Writer(f, db_header_from_columns)
408                            f.close()
409
410                            # Replace #CHROM line with rel columns
411                            header_list = db_for_header.read_header_file(
412                                header_file=header_file_tmp
413                            )
414                            header_list[-1] = "\t".join(db_header_columns)
415
416                    except:
417
418                        log.warning(
419                            f"No header for file {input_file}. Set as default VCF header"
420                        )
421                        header_list = default_header_list
422
423            else:  # try for unknown format ?
424
425                log.error(f"Input file format '{input_format}' not available")
426                raise ValueError(f"Input file format '{input_format}' not available")
427
428            if not header_list:
429                header_list = default_header_list
430
431            # header as list
432            self.header_list = header_list
433
434            # header as VCF object
435            self.header_vcf = vcf.Reader(io.StringIO("\n".join(header_list)))
436
437        else:
438
439            self.header_list = None
440            self.header_vcf = None

It reads the header of a VCF file and stores it as a list of strings and as a VCF object

def get_query_to_df(self, query: str = '', limit: int = None) -> pandas.core.frame.DataFrame: View Source

442    def get_query_to_df(self, query: str = "", limit: int = None) -> pd.DataFrame:
443        """
444        The `get_query_to_df` function takes a query as a string and returns the result as a pandas
445        DataFrame based on the connection format.
446
447        :param query: The `query` parameter in the `get_query_to_df` function is a string that
448        represents the SQL query you want to execute. This query will be used to fetch data from a
449        database and convert it into a pandas DataFrame
450        :type query: str
451        :param limit: The `limit` parameter in the `get_query_to_df` function is used to specify the
452        maximum number of rows to be returned in the resulting dataframe. If a limit is provided, the
453        function will only fetch up to that number of rows from the database query result. If no limit
454        is specified,
455        :type limit: int
456        :return: A pandas DataFrame is being returned by the `get_query_to_df` function.
457        """
458
459        # Connexion format
460        connexion_format = self.get_connexion_format()
461
462        # Limit in query
463        if limit:
464            pd.set_option("display.max_rows", limit)
465            if connexion_format in ["duckdb"]:
466                df = (
467                    self.conn.execute(query)
468                    .fetch_record_batch(limit)
469                    .read_next_batch()
470                    .to_pandas()
471                )
472            elif connexion_format in ["sqlite"]:
473                df = next(pd.read_sql_query(query, self.conn, chunksize=limit))
474
475        # Full query
476        else:
477            if connexion_format in ["duckdb"]:
478                df = self.conn.execute(query).df()
479            elif connexion_format in ["sqlite"]:
480                df = pd.read_sql_query(query, self.conn)
481
482        return df

The get_query_to_df function takes a query as a string and returns the result as a pandas DataFrame based on the connection format.

Parameters

query: The query parameter in the get_query_to_df function is a string that represents the SQL query you want to execute. This query will be used to fetch data from a database and convert it into a pandas DataFrame
limit: The limit parameter in the get_query_to_df function is used to specify the maximum number of rows to be returned in the resulting dataframe. If a limit is provided, the function will only fetch up to that number of rows from the database query result. If no limit is specified,

Returns

A pandas DataFrame is being returned by the get_query_to_df function.

def get_overview(self) -> None: View Source

484    def get_overview(self) -> None:
485        """
486        The function prints the input, output, config, and dataframe of the current object
487        """
488        table_variants_from = self.get_table_variants(clause="from")
489        sql_columns = self.get_header_columns_as_sql()
490        sql_query_export = f"SELECT {sql_columns} FROM {table_variants_from}"
491        df = self.get_query_to_df(sql_query_export)
492        log.info(
493            "Input:  "
494            + str(self.get_input())
495            + " ["
496            + str(str(self.get_input_format()))
497            + "]"
498        )
499        log.info(
500            "Output: "
501            + str(self.get_output())
502            + " ["
503            + str(str(self.get_output_format()))
504            + "]"
505        )
506        log.info("Config: ")
507        for d in str(json.dumps(self.get_config(), indent=4, sort_keys=True)).split(
508            "\n"
509        ):
510            log.info("\t" + str(d))
511        log.info("Param: ")
512        for d in str(json.dumps(self.get_param(), indent=4, sort_keys=True)).split(
513            "\n"
514        ):
515            log.info("\t" + str(d))
516        log.info("Sample list: " + str(self.get_header_sample_list()))
517        log.info("Dataframe: ")
518        for d in str(df).split("\n"):
519            log.info("\t" + str(d))
520
521        # garbage collector
522        del df
523        gc.collect()
524
525        return None

The function prints the input, output, config, and dataframe of the current object

def get_stats(self) -> dict: View Source

527    def get_stats(self) -> dict:
528        """
529        The `get_stats` function calculates and returns various statistics of the current object,
530        including information about the input file, variants, samples, header fields, quality, and
531        SNVs/InDels.
532        :return: a dictionary containing various statistics of the current object. The dictionary has
533        the following structure:
534        """
535
536        # Log
537        log.info(f"Stats Calculation...")
538
539        # table varaints
540        table_variants_from = self.get_table_variants()
541
542        # stats dict
543        stats = {"Infos": {}}
544
545        ### File
546        input_file = self.get_input()
547        stats["Infos"]["Input file"] = input_file
548
549        # Header
550        header_infos = self.get_header().infos
551        header_formats = self.get_header().formats
552        header_infos_list = list(header_infos)
553        header_formats_list = list(header_formats)
554
555        ### Variants
556
557        stats["Variants"] = {}
558
559        # Variants by chr
560        sql_query_nb_variant_by_chrom = f'SELECT "#CHROM" as CHROM, count(*) as count FROM {table_variants_from} GROUP BY "#CHROM"'
561        df_nb_of_variants_by_chrom = self.get_query_to_df(sql_query_nb_variant_by_chrom)
562        nb_of_variants_by_chrom = df_nb_of_variants_by_chrom.sort_values(
563            by=["CHROM"], kind="quicksort"
564        )
565
566        # Total number of variants
567        nb_of_variants = nb_of_variants_by_chrom["count"].sum()
568
569        # Calculate percentage
570        nb_of_variants_by_chrom["percent"] = nb_of_variants_by_chrom["count"].apply(
571            lambda x: (x / nb_of_variants)
572        )
573
574        stats["Variants"]["Number of variants by chromosome"] = (
575            nb_of_variants_by_chrom.to_dict(orient="index")
576        )
577
578        stats["Infos"]["Number of variants"] = int(nb_of_variants)
579
580        ### Samples
581
582        # Init
583        samples = {}
584        nb_of_samples = 0
585
586        # Check Samples
587        if "GT" in header_formats_list and "FORMAT" in self.get_header_columns():
588            log.debug(f"Check samples...")
589            for sample in self.get_header_sample_list():
590                sql_query_samples = f"""
591                    SELECT  '{sample}' as sample,
592                            REGEXP_EXTRACT("{sample}", '^([0-9/|.]*)[:]*',1) as genotype,
593                            count(REGEXP_EXTRACT("{sample}", '^([0-9/|.]*)[:]*',1)) as count,
594                            concat((count(REGEXP_EXTRACT("{sample}", '^([0-9/|.]*)[:]*',1))/{nb_of_variants})) as percentage
595                    FROM {table_variants_from}
596                    WHERE (
597                        regexp_matches("{sample}", '^[0-9]([/|][0-9])+')
598                        AND
599                        len(string_split(CAST("FORMAT" AS VARCHAR), ':')) = len(string_split(CAST("{sample}" AS VARCHAR), ':'))
600                      )
601                    GROUP BY genotype
602                    """
603                sql_query_genotype_df = self.conn.execute(sql_query_samples).df()
604                sample_genotype_count = sql_query_genotype_df["count"].sum()
605                if len(sql_query_genotype_df):
606                    nb_of_samples += 1
607                    samples[f"{sample} - {sample_genotype_count} variants"] = (
608                        sql_query_genotype_df.to_dict(orient="index")
609                    )
610
611            stats["Samples"] = samples
612            stats["Infos"]["Number of samples"] = nb_of_samples
613
614        # #
615        # if "FORMAT" in self.get_header_columns() and "DP" in header_formats_list:
616        #     stats["Infos"]["Number of samples"] = nb_of_samples
617        # elif nb_of_samples:
618        #     stats["Infos"]["Number of samples"] = "not a VCF format"
619
620        ### INFO and FORMAT fields
621        header_types_df = {}
622        header_types_list = {
623            "List of INFO fields": header_infos,
624            "List of FORMAT fields": header_formats,
625        }
626        i = 0
627        for header_type in header_types_list:
628
629            header_type_infos = header_types_list.get(header_type)
630            header_infos_dict = {}
631
632            for info in header_type_infos:
633
634                i += 1
635                header_infos_dict[i] = {}
636
637                # ID
638                header_infos_dict[i]["id"] = info
639
640                # num
641                genotype_map = {None: ".", -1: "A", -2: "G", -3: "R"}
642                if header_type_infos[info].num in genotype_map.keys():
643                    header_infos_dict[i]["Number"] = genotype_map.get(
644                        header_type_infos[info].num
645                    )
646                else:
647                    header_infos_dict[i]["Number"] = header_type_infos[info].num
648
649                # type
650                if header_type_infos[info].type:
651                    header_infos_dict[i]["Type"] = header_type_infos[info].type
652                else:
653                    header_infos_dict[i]["Type"] = "."
654
655                # desc
656                if header_type_infos[info].desc != None:
657                    header_infos_dict[i]["Description"] = header_type_infos[info].desc
658                else:
659                    header_infos_dict[i]["Description"] = ""
660
661            if len(header_infos_dict):
662                header_types_df[header_type] = pd.DataFrame.from_dict(
663                    header_infos_dict, orient="index"
664                ).to_dict(orient="index")
665
666        # Stats
667        stats["Infos"]["Number of INFO fields"] = len(header_infos_list)
668        stats["Infos"]["Number of FORMAT fields"] = len(header_formats_list)
669        stats["Header"] = header_types_df
670
671        ### QUAL
672        if "QUAL" in self.get_header_columns():
673            sql_query_qual = f"""
674                    SELECT
675                        avg(CAST(QUAL AS INTEGER)) AS Average,
676                        min(CAST(QUAL AS INTEGER)) AS Minimum,
677                        max(CAST(QUAL AS INTEGER)) AS Maximum,
678                        stddev(CAST(QUAL AS INTEGER)) AS StandardDeviation,
679                        median(CAST(QUAL AS INTEGER)) AS Median,
680                        variance(CAST(QUAL AS INTEGER)) AS Variance
681                    FROM {table_variants_from}
682                    WHERE CAST(QUAL AS VARCHAR) NOT IN ('.')
683                    """
684
685            qual = self.conn.execute(sql_query_qual).df().to_dict(orient="index")
686            stats["Quality"] = {"Stats": qual}
687
688        ### SNV and InDel
689
690        sql_query_snv = f"""
691            
692            SELECT Type, count FROM (
693
694                    SELECT
695                        'Total' AS Type,
696                        count(*) AS count
697                    FROM {table_variants_from}
698
699                    UNION
700
701                    SELECT
702                        'MNV' AS Type,
703                        count(*) AS count
704                    FROM {table_variants_from}
705                    WHERE len(REF) > 1 AND len(ALT) > 1
706                    AND len(REF) = len(ALT)
707
708                    UNION
709
710                    SELECT
711                        'InDel' AS Type,
712                        count(*) AS count
713                    FROM {table_variants_from}
714                    WHERE len(REF) > 1 OR len(ALT) > 1
715                    AND len(REF) != len(ALT)
716                    
717                    UNION
718
719                    SELECT
720                        'SNV' AS Type,
721                        count(*) AS count
722                    FROM {table_variants_from}
723                    WHERE len(REF) = 1 AND len(ALT) = 1
724
725                )
726
727            ORDER BY count DESC
728
729                """
730        snv_indel = self.conn.execute(sql_query_snv).df().to_dict(orient="index")
731
732        sql_query_snv_substitution = f"""
733                SELECT
734                    concat(REF, '>', ALT) AS 'Substitution',
735                    count(*) AS count
736                FROM {table_variants_from}
737                WHERE len(REF) = 1 AND len(ALT) = 1
738                GROUP BY REF, ALT
739                ORDER BY count(*) DESC
740                """
741        snv_substitution = (
742            self.conn.execute(sql_query_snv_substitution).df().to_dict(orient="index")
743        )
744        stats["Variants"]["Counts"] = snv_indel
745        stats["Variants"]["Substitutions"] = snv_substitution
746
747        return stats

The get_stats function calculates and returns various statistics of the current object, including information about the input file, variants, samples, header fields, quality, and SNVs/InDels.

Returns

a dictionary containing various statistics of the current object. The dictionary has the following structure:

def stats_to_file(self, file: str = None) -> str: View Source

749    def stats_to_file(self, file: str = None) -> str:
750        """
751        The function `stats_to_file` takes a file name as input, retrieves statistics, serializes them
752        into a JSON object, and writes the JSON object to the specified file.
753
754        :param file: The `file` parameter is a string that represents the file path where the JSON data
755        will be written
756        :type file: str
757        :return: the name of the file that was written to.
758        """
759
760        # Get stats
761        stats = self.get_stats()
762
763        # Serializing json
764        json_object = json.dumps(stats, indent=4)
765
766        # Writing to sample.json
767        with open(file, "w") as outfile:
768            outfile.write(json_object)
769
770        return file

The function stats_to_file takes a file name as input, retrieves statistics, serializes them into a JSON object, and writes the JSON object to the specified file.

Parameters

file: The file parameter is a string that represents the file path where the JSON data will be written

Returns

the name of the file that was written to.

def print_stats(self, output_file: str = None, json_file: str = None) -> None: View Source

772    def print_stats(self, output_file: str = None, json_file: str = None) -> None:
773        """
774        The `print_stats` function generates a markdown file and prints the statistics contained in a
775        JSON file in a formatted manner.
776
777        :param output_file: The `output_file` parameter is a string that specifies the path and filename
778        of the output file where the stats will be printed in Markdown format. If no `output_file` is
779        provided, a temporary directory will be created and the stats will be saved in a file named
780        "stats.md" within that
781        :type output_file: str
782        :param json_file: The `json_file` parameter is a string that represents the path to the JSON
783        file where the statistics will be saved. If no value is provided, a temporary directory will be
784        created and a default file name "stats.json" will be used
785        :type json_file: str
786        :return: The function `print_stats` does not return any value. It has a return type annotation
787        of `None`.
788        """
789
790        # Full path
791        output_file = full_path(output_file)
792        json_file = full_path(json_file)
793
794        with tempfile.TemporaryDirectory() as tmpdir:
795
796            # Files
797            if not output_file:
798                output_file = os.path.join(tmpdir, "stats.md")
799            if not json_file:
800                json_file = os.path.join(tmpdir, "stats.json")
801
802            # Create folders
803            if not os.path.exists(os.path.dirname(output_file)):
804                Path(os.path.dirname(output_file)).mkdir(parents=True, exist_ok=True)
805            if not os.path.exists(os.path.dirname(json_file)):
806                Path(os.path.dirname(json_file)).mkdir(parents=True, exist_ok=True)
807
808            # Create stats JSON file
809            stats_file = self.stats_to_file(file=json_file)
810
811            # Print stats file
812            with open(stats_file) as f:
813                stats = yaml.safe_load(f)
814
815            # Output
816            output_title = []
817            output_index = []
818            output = []
819
820            # Title
821            output_title.append("# HOWARD Stats")
822
823            # Index
824            output_index.append("## Index")
825
826            # Process sections
827            for section in stats:
828                infos = stats.get(section)
829                section_link = "#" + section.lower().replace(" ", "-")
830                output.append(f"## {section}")
831                output_index.append(f"- [{section}]({section_link})")
832
833                if len(infos):
834                    for info in infos:
835                        try:
836                            df = pd.DataFrame.from_dict(infos.get(info), orient="index")
837                            is_df = True
838                        except:
839                            try:
840                                df = pd.DataFrame.from_dict(
841                                    json.loads((infos.get(info))), orient="index"
842                                )
843                                is_df = True
844                            except:
845                                is_df = False
846                        if is_df:
847                            output.append(f"### {info}")
848                            info_link = "#" + info.lower().replace(" ", "-")
849                            output_index.append(f"   - [{info}]({info_link})")
850                            output.append(f"{df.to_markdown(index=False)}")
851                        else:
852                            output.append(f"- {info}: {infos.get(info)}")
853                else:
854                    output.append(f"NA")
855
856            # Write stats in markdown file
857            with open(output_file, "w") as fp:
858                for item in output_title:
859                    fp.write("%s\n" % item)
860                for item in output_index:
861                    fp.write("%s\n" % item)
862                for item in output:
863                    fp.write("%s\n" % item)
864
865            # Output stats in markdown
866            print("")
867            print("\n\n".join(output_title))
868            print("")
869            print("\n\n".join(output))
870            print("")
871
872        return None

The print_stats function generates a markdown file and prints the statistics contained in a JSON file in a formatted manner.

Parameters

output_file: The output_file parameter is a string that specifies the path and filename of the output file where the stats will be printed in Markdown format. If no output_file is provided, a temporary directory will be created and the stats will be saved in a file named "stats.md" within that
json_file: The json_file parameter is a string that represents the path to the JSON file where the statistics will be saved. If no value is provided, a temporary directory will be created and a default file name "stats.json" will be used

Returns

The function print_stats does not return any value. It has a return type annotation of None.

def get_input(self) -> str: View Source

874    def get_input(self) -> str:
875        """
876        It returns the value of the input variable.
877        :return: The input is being returned.
878        """
879        return self.input

It returns the value of the input variable.

Returns

The input is being returned.

def get_input_format(self, input_file: str = None) -> str: View Source

881    def get_input_format(self, input_file: str = None) -> str:
882        """
883        This function returns the format of the input variable, either from the provided input file or
884        by prompting for input.
885
886        :param input_file: The `input_file` parameter in the `get_input_format` method is a string that
887        represents the file path of the input file. If no `input_file` is provided when calling the
888        method, it will default to `None`
889        :type input_file: str
890        :return: The format of the input variable is being returned.
891        """
892
893        if not input_file:
894            input_file = self.get_input()
895        input_format = get_file_format(input_file)
896        return input_format

This function returns the format of the input variable, either from the provided input file or by prompting for input.

Parameters

input_file: The input_file parameter in the get_input_format method is a string that represents the file path of the input file. If no input_file is provided when calling the method, it will default to None

Returns

The format of the input variable is being returned.

def get_input_compressed(self, input_file: str = None) -> str: View Source

898    def get_input_compressed(self, input_file: str = None) -> str:
899        """
900        The function `get_input_compressed` returns the format of the input variable after compressing
901        it.
902
903        :param input_file: The `input_file` parameter in the `get_input_compressed` method is a string
904        that represents the file path of the input file. If no `input_file` is provided when calling the
905        method, it will default to `None` and the method will then call `self.get_input()` to
906        :type input_file: str
907        :return: The function `get_input_compressed` returns the compressed format of the input
908        variable.
909        """
910
911        if not input_file:
912            input_file = self.get_input()
913        input_compressed = get_file_compressed(input_file)
914        return input_compressed

The function get_input_compressed returns the format of the input variable after compressing it.

Parameters

input_file: The input_file parameter in the get_input_compressed method is a string that represents the file path of the input file. If no input_file is provided when calling the method, it will default to None and the method will then call self.get_input() to

Returns

The function get_input_compressed returns the compressed format of the input variable.

def get_output(self) -> str: View Source

916    def get_output(self) -> str:
917        """
918        It returns the output of the neuron.
919        :return: The output of the neural network.
920        """
921
922        return self.output

It returns the output of the neuron.

Returns

The output of the neural network.

def get_output_format(self, output_file: str = None) -> str: View Source

924    def get_output_format(self, output_file: str = None) -> str:
925        """
926        The function `get_output_format` returns the format of the input variable or the output file if
927        provided.
928
929        :param output_file: The `output_file` parameter in the `get_output_format` method is a string
930        that represents the file path of the output file. If no `output_file` is provided when calling
931        the method, it will default to the output obtained from the `get_output` method of the class
932        instance. The
933        :type output_file: str
934        :return: The format of the input variable is being returned.
935        """
936
937        if not output_file:
938            output_file = self.get_output()
939        output_format = get_file_format(output_file)
940
941        return output_format

The function get_output_format returns the format of the input variable or the output file if provided.

Parameters

output_file: The output_file parameter in the get_output_format method is a string that represents the file path of the output file. If no output_file is provided when calling the method, it will default to the output obtained from the get_output method of the class instance. The

Returns

The format of the input variable is being returned.

def get_config(self) -> dict: View Source

943    def get_config(self) -> dict:
944        """
945        It returns the config
946        :return: The config variable is being returned.
947        """
948        return self.config

It returns the config

Returns

The config variable is being returned.

def get_param(self) -> dict: View Source

950    def get_param(self) -> dict:
951        """
952        It returns the param
953        :return: The param variable is being returned.
954        """
955        return self.param

It returns the param

Returns

The param variable is being returned.

def get_connexion_db(self) -> str: View Source

957    def get_connexion_db(self) -> str:
958        """
959        It returns the connexion_db attribute of the object
960        :return: The connexion_db is being returned.
961        """
962        return self.connexion_db

It returns the connexion_db attribute of the object

Returns

The connexion_db is being returned.

def get_prefix(self) -> str: View Source

964    def get_prefix(self) -> str:
965        """
966        It returns the prefix of the object.
967        :return: The prefix is being returned.
968        """
969        return self.prefix

It returns the prefix of the object.

Returns

The prefix is being returned.

def get_table_variants(self, clause: str = 'select') -> str: View Source

971    def get_table_variants(self, clause: str = "select") -> str:
972        """
973        This function returns the table_variants attribute of the object
974
975        :param clause: the type of clause the table will be used. Either "select" or "from" (optional),
976        defaults to select (optional)
977        :return: The table_variants attribute of the object.
978        """
979
980        # Access
981        access = self.get_config().get("access", None)
982
983        # Clauses "select", "where", "update"
984        if clause in ["select", "where", "update"]:
985            table_variants = self.table_variants
986        # Clause "from"
987        elif clause in ["from"]:
988            # For Read Only
989            if self.get_input_format() in ["parquet"] and access in ["RO"]:
990                input_file = self.get_input()
991                table_variants = f"'{input_file}' as variants"
992            # For Read Write
993            else:
994                table_variants = f"{self.table_variants} as variants"
995        else:
996            table_variants = self.table_variants
997        return table_variants

This function returns the table_variants attribute of the object

Parameters

clause: the type of clause the table will be used. Either "select" or "from" (optional), defaults to select (optional)

Returns

The table_variants attribute of the object.

def get_tmp_dir(self) -> str: View Source

 999    def get_tmp_dir(self) -> str:
1000        """
1001        The function `get_tmp_dir` returns the temporary directory path based on configuration
1002        parameters or a default path.
1003        :return: The `get_tmp_dir` method is returning the temporary directory path based on the
1004        configuration, parameters, and a default value of "/tmp".
1005        """
1006
1007        return get_tmp(
1008            config=self.get_config(), param=self.get_param(), default_tmp="/tmp"
1009        )

The function get_tmp_dir returns the temporary directory path based on configuration parameters or a default path.

Returns

The get_tmp_dir method is returning the temporary directory path based on the configuration, parameters, and a default value of "/tmp".

def get_connexion_type(self) -> str: View Source

1011    def get_connexion_type(self) -> str:
1012        """
1013        If the connexion type is not in the list of allowed connexion types, raise a ValueError
1014
1015        :return: The connexion type is being returned.
1016        """
1017        return self.get_config().get("connexion_type", "memory")

If the connexion type is not in the list of allowed connexion types, raise a ValueError

Returns

The connexion type is being returned.

def get_connexion(self): View Source

1019    def get_connexion(self):
1020        """
1021        It returns the connection object
1022
1023        :return: The connection object.
1024        """
1025        return self.conn

It returns the connection object

Returns

The connection object.

def close_connexion(self) -> None: View Source

1027    def close_connexion(self) -> None:
1028        """
1029        This function closes the connection to the database.
1030        :return: The connection is being closed.
1031        """
1032        return self.conn.close()

This function closes the connection to the database.

Returns

The connection is being closed.

def get_header(self, type: str = 'vcf'): View Source

1034    def get_header(self, type: str = "vcf"):
1035        """
1036        This function returns the header of the VCF file as a list of strings
1037
1038        :param type: the type of header you want to get, defaults to vcf (optional)
1039        :return: The header of the vcf file.
1040        """
1041
1042        if self.header_vcf:
1043            if type == "vcf":
1044                return self.header_vcf
1045            elif type == "list":
1046                return self.header_list
1047        else:
1048            if type == "vcf":
1049                header = vcf.Reader(io.StringIO("\n".join(vcf_required)))
1050                return header
1051            elif type == "list":
1052                return vcf_required

This function returns the header of the VCF file as a list of strings

Parameters

type: the type of header you want to get, defaults to vcf (optional)

Returns

The header of the vcf file.

def get_header_length(self, file: str = None) -> int: View Source

1054    def get_header_length(self, file: str = None) -> int:
1055        """
1056        The function `get_header_length` returns the length of the header list, excluding the #CHROM
1057        line.
1058
1059        :param file: The `file` parameter is an optional argument that specifies the path to a VCF
1060        header file. If this argument is provided, the function will read the header from the specified
1061        file and return the length of the header list minus 1 (to exclude the #CHROM line)
1062        :type file: str
1063        :return: the length of the header list, excluding the #CHROM line.
1064        """
1065
1066        if file:
1067            return len(self.read_vcf_header_file(file=file)) - 1
1068        elif self.get_header(type="list"):
1069            return len(self.get_header(type="list")) - 1
1070        else:
1071            return 0

The function get_header_length returns the length of the header list, excluding the #CHROM line.

Parameters

file: The file parameter is an optional argument that specifies the path to a VCF header file. If this argument is provided, the function will read the header from the specified file and return the length of the header list minus 1 (to exclude the #CHROM line)

Returns

the length of the header list, excluding the #CHROM line.

def get_header_columns(self) -> str: View Source

1073    def get_header_columns(self) -> str:
1074        """
1075        This function returns the header list of a VCF
1076
1077        :return: The length of the header list.
1078        """
1079        if self.get_header():
1080            return self.get_header(type="list")[-1]
1081        else:
1082            return ""

This function returns the header list of a VCF

Returns

The length of the header list.

def get_header_columns_as_list(self) -> list: View Source

1084    def get_header_columns_as_list(self) -> list:
1085        """
1086        This function returns the header list of a VCF
1087
1088        :return: The length of the header list.
1089        """
1090        if self.get_header():
1091            return self.get_header_columns().strip().split("\t")
1092        else:
1093            return []

This function returns the header list of a VCF

Returns

The length of the header list.

def get_header_columns_as_sql(self) -> str: View Source

1095    def get_header_columns_as_sql(self) -> str:
1096        """
1097        This function retruns header length (without #CHROM line)
1098
1099        :return: The length of the header list.
1100        """
1101        sql_column_list = []
1102        for col in self.get_header_columns_as_list():
1103            sql_column_list.append(f'"{col}"')
1104        return ",".join(sql_column_list)

This function retruns header length (without #CHROM line)

Returns

The length of the header list.

def get_header_sample_list(self) -> list: View Source

1106    def get_header_sample_list(self) -> list:
1107        """
1108        This function retruns header length (without #CHROM line)
1109
1110        :return: The length of the header list.
1111        """
1112        return self.header_vcf.samples

This function retruns header length (without #CHROM line)

Returns

The length of the header list.

def get_verbose(self) -> bool: View Source

1114    def get_verbose(self) -> bool:
1115        """
1116        It returns the value of the "verbose" key in the config dictionary, or False if the key doesn't
1117        exist
1118
1119        :return: The value of the key "verbose" in the config dictionary.
1120        """
1121        return self.get_config().get("verbose", False)

It returns the value of the "verbose" key in the config dictionary, or False if the key doesn't exist

Returns

The value of the key "verbose" in the config dictionary.

def get_connexion_format(self) -> str: View Source

1123    def get_connexion_format(self) -> str:
1124        """
1125        It returns the connexion format of the object.
1126        :return: The connexion_format is being returned.
1127        """
1128        connexion_format = self.connexion_format
1129        if connexion_format not in ["duckdb", "sqlite"]:
1130            log.error(f"Unknown connexion format {connexion_format}")
1131            raise ValueError(f"Unknown connexion format {connexion_format}")
1132        else:
1133            return connexion_format

It returns the connexion format of the object.

Returns

The connexion_format is being returned.

def insert_file_to_table( self, file, columns: str, header_len: int = 0, sep: str = '\t', chunksize: int = 1000000) -> None: View Source

1135    def insert_file_to_table(
1136        self,
1137        file,
1138        columns: str,
1139        header_len: int = 0,
1140        sep: str = "\t",
1141        chunksize: int = 1000000,
1142    ) -> None:
1143        """
1144        The function reads a file in chunks and inserts each chunk into a table based on the specified
1145        database format.
1146
1147        :param file: The `file` parameter is the file that you want to load into a table. It should be
1148        the path to the file on your system
1149        :param columns: The `columns` parameter in the `insert_file_to_table` function is a string that
1150        should contain the names of the columns in the table where the data will be inserted. The column
1151        names should be separated by commas within the string. For example, if you have columns named
1152        "id", "name
1153        :type columns: str
1154        :param header_len: The `header_len` parameter in the `insert_file_to_table` function specifies
1155        the number of lines to skip at the beginning of the file before reading the actual data. This
1156        parameter allows you to skip any header information present in the file before processing the
1157        data, defaults to 0
1158        :type header_len: int (optional)
1159        :param sep: The `sep` parameter in the `insert_file_to_table` function is used to specify the
1160        separator character that is used in the file being read. In this case, the default separator is
1161        set to `\t`, which represents a tab character. You can change this parameter to a different
1162        separator character if, defaults to \t
1163        :type sep: str (optional)
1164        :param chunksize: The `chunksize` parameter specifies the number of rows to read in at a time
1165        when processing the file in chunks. In the provided code snippet, the default value for
1166        `chunksize` is set to 1000000. This means that the file will be read in chunks of 1,, defaults
1167        to 1000000
1168        :type chunksize: int (optional)
1169        """
1170
1171        # Config
1172        chunksize = self.get_config().get("load", {}).get("chunk", chunksize)
1173        connexion_format = self.get_connexion_format()
1174
1175        log.debug("chunksize: " + str(chunksize))
1176
1177        if chunksize:
1178            for chunk in pd.read_csv(
1179                file, skiprows=header_len, sep=sep, chunksize=chunksize, engine="c"
1180            ):
1181                if connexion_format in ["duckdb"]:
1182                    sql_insert_into = (
1183                        f"INSERT INTO variants ({columns}) SELECT {columns} FROM chunk"
1184                    )
1185                    self.conn.execute(sql_insert_into)
1186                elif connexion_format in ["sqlite"]:
1187                    chunk.to_sql("variants", self.conn, if_exists="append", index=False)

The function reads a file in chunks and inserts each chunk into a table based on the specified database format.

Parameters

file: The file parameter is the file that you want to load into a table. It should be the path to the file on your system
columns: The columns parameter in the insert_file_to_table function is a string that should contain the names of the columns in the table where the data will be inserted. The column names should be separated by commas within the string. For example, if you have columns named "id", "name
header_len: The header_len parameter in the insert_file_to_table function specifies the number of lines to skip at the beginning of the file before reading the actual data. This parameter allows you to skip any header information present in the file before processing the data, defaults to 0
sep: The sep parameter in the insert_file_to_table function is used to specify the separator character that is used in the file being read. In this case, the default separator is set to , which represents a tab character. You can change this parameter to a different separator character if, defaults to
chunksize: The chunksize parameter specifies the number of rows to read in at a time when processing the file in chunks. In the provided code snippet, the default value for chunksize is set to 1000000. This means that the file will be read in chunks of 1,, defaults to 1000000

def load_data( self, input_file: str = None, drop_variants_table: bool = False, sample_size: int = 20480) -> None: View Source

1189    def load_data(
1190        self,
1191        input_file: str = None,
1192        drop_variants_table: bool = False,
1193        sample_size: int = 20480,
1194    ) -> None:
1195        """
1196        The `load_data` function reads a VCF file and inserts it into a table, with options to drop the
1197        table before loading the data and specify a sample size.
1198
1199        :param input_file: The path to the input file. This is the VCF file that will be loaded into the
1200        table
1201        :type input_file: str
1202        :param drop_variants_table: The `drop_variants_table` parameter is a boolean flag that
1203        determines whether the variants table should be dropped before loading the data. If set to
1204        `True`, the variants table will be dropped. If set to `False` (default), the variants table will
1205        not be dropped, defaults to False
1206        :type drop_variants_table: bool (optional)
1207        :param sample_size: The `sample_size` parameter determines the number of rows to be sampled from
1208        the input file. If it is set to `None`, the default value of 20480 will be used, defaults to
1209        20480
1210        :type sample_size: int (optional)
1211        """
1212
1213        log.info("Loading...")
1214
1215        # change input file
1216        if input_file:
1217            self.set_input(input_file)
1218            self.set_header()
1219
1220        # drop variants table
1221        if drop_variants_table:
1222            self.drop_variants_table()
1223
1224        # get table variants
1225        table_variants = self.get_table_variants()
1226
1227        # Access
1228        access = self.get_config().get("access", None)
1229        log.debug(f"access: {access}")
1230
1231        # Input format and compress
1232        input_format = self.get_input_format()
1233        input_compressed = self.get_input_compressed()
1234        log.debug(f"input_format: {input_format}")
1235        log.debug(f"input_compressed: {input_compressed}")
1236
1237        # input_compressed_format
1238        if input_compressed:
1239            input_compressed_format = "gzip"
1240        else:
1241            input_compressed_format = "none"
1242        log.debug(f"input_compressed_format: {input_compressed_format}")
1243
1244        # Connexion format
1245        connexion_format = self.get_connexion_format()
1246
1247        # Sample size
1248        if not sample_size:
1249            sample_size = -1
1250        log.debug(f"sample_size: {sample_size}")
1251
1252        # Load data
1253        log.debug(f"Load Data from {input_format}")
1254
1255        # DuckDB connexion
1256        if connexion_format in ["duckdb"]:
1257
1258            # Database already exists
1259            if self.input_format in ["db", "duckdb"]:
1260
1261                if connexion_format in ["duckdb"]:
1262                    log.debug(f"Input file format '{self.input_format}' duckDB")
1263                else:
1264                    log.error(
1265                        f"Input file format '{self.input_format}' not compatilbe with database format '{connexion_format}'"
1266                    )
1267                    raise ValueError(
1268                        f"Input file format '{self.input_format}' not compatilbe with database format '{connexion_format}'"
1269                    )
1270
1271            # Load from existing database format
1272            else:
1273
1274                try:
1275                    # Create Table or View
1276                    database = Database(database=self.input)
1277                    sql_from = database.get_sql_from(sample_size=sample_size)
1278
1279                    if access in ["RO"]:
1280                        sql_load = (
1281                            f"CREATE VIEW {table_variants} AS SELECT * FROM {sql_from}"
1282                        )
1283                    else:
1284                        sql_load = (
1285                            f"CREATE TABLE {table_variants} AS SELECT * FROM {sql_from}"
1286                        )
1287                    self.conn.execute(sql_load)
1288
1289                except:
1290                    # Format not available
1291                    log.error(f"Input file format '{self.input_format}' not available")
1292                    raise ValueError(
1293                        f"Input file format '{self.input_format}' not available"
1294                    )
1295
1296        # SQLite connexion
1297        elif connexion_format in ["sqlite"] and input_format in [
1298            "vcf",
1299            "tsv",
1300            "csv",
1301            "psv",
1302        ]:
1303
1304            # Main structure
1305            structure = {
1306                "#CHROM": "VARCHAR",
1307                "POS": "INTEGER",
1308                "ID": "VARCHAR",
1309                "REF": "VARCHAR",
1310                "ALT": "VARCHAR",
1311                "QUAL": "VARCHAR",
1312                "FILTER": "VARCHAR",
1313                "INFO": "VARCHAR",
1314            }
1315
1316            # Strcuture with samples
1317            structure_complete = structure
1318            if self.get_header_sample_list():
1319                structure["FORMAT"] = "VARCHAR"
1320                for sample in self.get_header_sample_list():
1321                    structure_complete[sample] = "VARCHAR"
1322
1323            # Columns list for create and insert
1324            sql_create_table_columns = []
1325            sql_create_table_columns_list = []
1326            for column in structure_complete:
1327                column_type = structure_complete[column]
1328                sql_create_table_columns.append(
1329                    f'"{column}" {column_type} default NULL'
1330                )
1331                sql_create_table_columns_list.append(f'"{column}"')
1332
1333            # Create database
1334            log.debug(f"Create Table {table_variants}")
1335            sql_create_table_columns_sql = ", ".join(sql_create_table_columns)
1336            sql_create_table_columns_list_sql = ", ".join(sql_create_table_columns_list)
1337            sql_create_table = f"CREATE TABLE IF NOT EXISTS {table_variants} ({sql_create_table_columns_sql})"
1338            self.conn.execute(sql_create_table)
1339
1340            # chunksize define length of file chunk load file
1341            chunksize = 100000
1342
1343            # delimiter
1344            delimiter = file_format_delimiters.get(input_format, "\t")
1345
1346            # Load the input file
1347            with open(self.input, "rt") as input_file:
1348
1349                # Use the appropriate file handler based on the input format
1350                if input_compressed:
1351                    input_file = bgzf.open(self.input, "rt")
1352                if input_format in ["vcf"]:
1353                    header_len = self.get_header_length()
1354                else:
1355                    header_len = 0
1356
1357                # Insert the file contents into a table
1358                self.insert_file_to_table(
1359                    input_file,
1360                    columns=sql_create_table_columns_list_sql,
1361                    header_len=header_len,
1362                    sep=delimiter,
1363                    chunksize=chunksize,
1364                )
1365
1366        else:
1367            log.error(
1368                f"Connexion format '{connexion_format}' not available with format '{input_format}'"
1369            )
1370            raise ValueError(
1371                f"Connexion format '{connexion_format}' not available with format '{input_format}'"
1372            )
1373
1374        # Explode INFOS fields into table fields
1375        if self.get_explode_infos():
1376            self.explode_infos(
1377                prefix=self.get_explode_infos_prefix(),
1378                fields=self.get_explode_infos_fields(),
1379                force=True,
1380            )
1381
1382        # Create index after insertion
1383        self.create_indexes()

The load_data function reads a VCF file and inserts it into a table, with options to drop the table before loading the data and specify a sample size.

Parameters

input_file: The path to the input file. This is the VCF file that will be loaded into the table
drop_variants_table: The drop_variants_table parameter is a boolean flag that determines whether the variants table should be dropped before loading the data. If set to True, the variants table will be dropped. If set to False (default), the variants table will not be dropped, defaults to False
sample_size: The sample_size parameter determines the number of rows to be sampled from the input file. If it is set to None, the default value of 20480 will be used, defaults to 20480

def get_explode_infos(self) -> bool: View Source

1385    def get_explode_infos(self) -> bool:
1386        """
1387        The function `get_explode_infos` returns the value of the "explode_infos" parameter, defaulting
1388        to False if it is not set.
1389        :return: The method is returning the value of the "explode_infos" parameter, which is a boolean
1390        value. If the parameter is not present, it will return False.
1391        """
1392
1393        return self.get_param().get("explode", {}).get("explode_infos", False)

The function get_explode_infos returns the value of the "explode_infos" parameter, defaulting to False if it is not set.

Returns

The method is returning the value of the "explode_infos" parameter, which is a boolean value. If the parameter is not present, it will return False.

def get_explode_infos_fields( self, explode_infos_fields: str = None, remove_fields_not_in_header: bool = False) -> list: View Source

1395    def get_explode_infos_fields(
1396        self,
1397        explode_infos_fields: str = None,
1398        remove_fields_not_in_header: bool = False,
1399    ) -> list:
1400        """
1401        The `get_explode_infos_fields` function returns a list of exploded information fields based on
1402        the input parameter `explode_infos_fields`.
1403
1404        :param explode_infos_fields: The `explode_infos_fields` parameter is a string that specifies the
1405        fields to be exploded. It can be set to "ALL" to explode all fields, or it can be a
1406        comma-separated list of field names to explode
1407        :type explode_infos_fields: str
1408        :param remove_fields_not_in_header: The parameter `remove_fields_not_in_header` is a boolean
1409        flag that determines whether to remove fields that are not present in the header. If it is set
1410        to `True`, any field that is not in the header will be excluded from the list of exploded
1411        information fields. If it is set to `, defaults to False
1412        :type remove_fields_not_in_header: bool (optional)
1413        :return: The function `get_explode_infos_fields` returns a list of exploded information fields.
1414        If the `explode_infos_fields` parameter is not provided or is set to None, it returns an empty
1415        list. If the parameter is provided and its value is "ALL", it also returns an empty list.
1416        Otherwise, it returns a list of exploded information fields after removing any spaces and
1417        splitting the string by commas.
1418        """
1419
1420        # If no fields, get it in param
1421        if not explode_infos_fields:
1422            explode_infos_fields = (
1423                self.get_param().get("explode", {}).get("explode_infos_fields", None)
1424            )
1425
1426        # If no fields, defined as all fields in header using keyword
1427        if not explode_infos_fields:
1428            explode_infos_fields = "*"
1429
1430        # If fields list not empty
1431        if explode_infos_fields:
1432
1433            # Input fields list
1434            if isinstance(explode_infos_fields, str):
1435                fields_input = explode_infos_fields.split(",")
1436            elif isinstance(explode_infos_fields, list):
1437                fields_input = explode_infos_fields
1438            else:
1439                fields_input = []
1440
1441            # Fields list without * keyword
1442            fields_without_all = fields_input.copy()
1443            if "*".casefold() in (item.casefold() for item in fields_without_all):
1444                fields_without_all.remove("*")
1445
1446            # Fields in header
1447            fields_in_header = sorted(list(set(self.get_header().infos)))
1448
1449            # Construct list of fields
1450            fields_output = []
1451            for field in fields_input:
1452
1453                # Strip field
1454                field = field.strip()
1455
1456                # format keyword * in regex
1457                if field.upper() in ["*"]:
1458                    field = ".*"
1459
1460                # Find all fields with pattern
1461                r = re.compile(field)
1462                fields_search = sorted(list(filter(r.match, fields_in_header)))
1463
1464                # Remove fields input from search
1465                if field in fields_search:
1466                    fields_search = [field]
1467                elif fields_search != [field]:
1468                    fields_search = sorted(
1469                        list(set(fields_search).difference(fields_input))
1470                    )
1471
1472                # If field is not in header (avoid not well formatted header)
1473                if not fields_search and not remove_fields_not_in_header:
1474                    fields_search = [field]
1475
1476                # Add found fields
1477                for new_field in fields_search:
1478                    # Add field, if not already exists, and if it is in header (if asked)
1479                    if (
1480                        new_field not in fields_output
1481                        and (
1482                            not remove_fields_not_in_header
1483                            or new_field in fields_in_header
1484                        )
1485                        and new_field not in [".*"]
1486                    ):
1487                        fields_output.append(new_field)
1488
1489            return fields_output
1490
1491        else:
1492
1493            return []

The get_explode_infos_fields function returns a list of exploded information fields based on the input parameter explode_infos_fields.

Parameters

explode_infos_fields: The explode_infos_fields parameter is a string that specifies the fields to be exploded. It can be set to "ALL" to explode all fields, or it can be a comma-separated list of field names to explode
remove_fields_not_in_header: The parameter remove_fields_not_in_header is a boolean flag that determines whether to remove fields that are not present in the header. If it is set to True, any field that is not in the header will be excluded from the list of exploded information fields. If it is set to `, defaults to False

Returns

The function get_explode_infos_fields returns a list of exploded information fields. If the explode_infos_fields parameter is not provided or is set to None, it returns an empty list. If the parameter is provided and its value is "ALL", it also returns an empty list. Otherwise, it returns a list of exploded information fields after removing any spaces and splitting the string by commas.

def get_explode_infos_prefix(self, explode_infos_prefix: str = None) -> str: View Source

1495    def get_explode_infos_prefix(self, explode_infos_prefix: str = None) -> str:
1496        """
1497        The function `get_explode_infos_prefix` returns the value of the `explode_infos_prefix` parameter, or
1498        the value of `self.get_param().get("explode_infos_prefix", None)` if `explode_infos_prefix` is
1499        not provided.
1500
1501        :param explode_infos_prefix: The parameter `explode_infos_prefix` is a string that specifies a
1502        prefix to be used for exploding or expanding information
1503        :type explode_infos_prefix: str
1504        :return: the value of the variable `explode_infos_prefix`.
1505        """
1506
1507        if not explode_infos_prefix:
1508            explode_infos_prefix = (
1509                self.get_param().get("explode", {}).get("explode_infos_prefix", "")
1510            )
1511
1512        return explode_infos_prefix

The function get_explode_infos_prefix returns the value of the explode_infos_prefix parameter, or the value of self.get_param().get("explode_infos_prefix", None) if explode_infos_prefix is not provided.

Parameters

explode_infos_prefix: The parameter explode_infos_prefix is a string that specifies a prefix to be used for exploding or expanding information

Returns

the value of the variable explode_infos_prefix.

def add_column( self, table_name, column_name, column_type, default_value=None, drop: bool = False) -> dict: View Source

1514    def add_column(
1515        self,
1516        table_name,
1517        column_name,
1518        column_type,
1519        default_value=None,
1520        drop: bool = False,
1521    ) -> dict:
1522        """
1523        The `add_column` function adds a column to a SQLite or DuckDB table with a default value if it
1524        doesn't already exist.
1525
1526        :param table_name: The name of the table to which you want to add a column
1527        :param column_name: The parameter "column_name" is the name of the column that you want to add
1528        to the table
1529        :param column_type: The `column_type` parameter specifies the data type of the column that you
1530        want to add to the table. It should be a string that represents the desired data type, such as
1531        "INTEGER", "TEXT", "REAL", etc
1532        :param default_value: The `default_value` parameter is an optional parameter that specifies the
1533        default value for the newly added column. If a default value is provided, it will be assigned to
1534        the column for any existing rows that do not have a value for that column
1535        :param drop: The `drop` parameter is a boolean flag that determines whether to drop the column
1536        if it already exists in the table. If `drop` is set to `True`, the function will drop the
1537        existing column before adding the new column. If `drop` is set to `False` (default),, defaults
1538        to False
1539        :type drop: bool (optional)
1540        :return: a boolean value indicating whether the column was successfully added to the table.
1541        """
1542
1543        # added
1544        added = False
1545        dropped = False
1546
1547        # Check if the column already exists in the table
1548        query = f""" SELECT * FROM {table_name} LIMIT 0 """
1549        columns = self.get_query_to_df(query).columns.tolist()
1550        if column_name.upper() in [c.upper() for c in columns]:
1551            log.debug(
1552                f"The {column_name} column already exists in the {table_name} table"
1553            )
1554            if drop:
1555                self.drop_column(table_name=table_name, column_name=column_name)
1556                dropped = True
1557            else:
1558                return None
1559        else:
1560            log.debug(f"The {column_name} column NOT exists in the {table_name} table")
1561
1562        # Add column in table
1563        add_column_query = (
1564            f""" ALTER TABLE {table_name} ADD COLUMN "{column_name}" {column_type} """
1565        )
1566        if default_value is not None:
1567            add_column_query += f" DEFAULT {default_value}"
1568        self.execute_query(add_column_query)
1569        added = not dropped
1570        log.debug(
1571            f"The {column_name} column was successfully added to the {table_name} table"
1572        )
1573
1574        if added:
1575            added_column = {
1576                "table_name": table_name,
1577                "column_name": column_name,
1578                "column_type": column_type,
1579                "default_value": default_value,
1580            }
1581        else:
1582            added_column = None
1583
1584        return added_column

The add_column function adds a column to a SQLite or DuckDB table with a default value if it doesn't already exist.

Parameters

table_name: The name of the table to which you want to add a column
column_name: The parameter "column_name" is the name of the column that you want to add to the table
column_type: The column_type parameter specifies the data type of the column that you want to add to the table. It should be a string that represents the desired data type, such as "INTEGER", "TEXT", "REAL", etc
default_value: The default_value parameter is an optional parameter that specifies the default value for the newly added column. If a default value is provided, it will be assigned to the column for any existing rows that do not have a value for that column
drop: The drop parameter is a boolean flag that determines whether to drop the column if it already exists in the table. If drop is set to True, the function will drop the existing column before adding the new column. If drop is set to False (default),, defaults to False

Returns

a boolean value indicating whether the column was successfully added to the table.

def drop_column( self, column: dict = None, table_name: str = None, column_name: str = None) -> bool: View Source

1586    def drop_column(
1587        self, column: dict = None, table_name: str = None, column_name: str = None
1588    ) -> bool:
1589        """
1590        The `drop_column` function drops a specified column from a given table in a database and returns
1591        True if the column was successfully dropped, and False if the column does not exist in the
1592        table.
1593
1594        :param column: The `column` parameter is a dictionary that contains information about the column
1595        you want to drop. It has two keys:
1596        :type column: dict
1597        :param table_name: The `table_name` parameter is the name of the table from which you want to
1598        drop a column
1599        :type table_name: str
1600        :param column_name: The `column_name` parameter is the name of the column that you want to drop
1601        from the table
1602        :type column_name: str
1603        :return: a boolean value. It returns True if the column was successfully dropped from the table,
1604        and False if the column does not exist in the table.
1605        """
1606
1607        # Find column infos
1608        if column:
1609            if isinstance(column, dict):
1610                table_name = column.get("table_name", None)
1611                column_name = column.get("column_name", None)
1612            elif isinstance(column, str):
1613                table_name = self.get_table_variants()
1614                column_name = column
1615            else:
1616                table_name = None
1617                column_name = None
1618
1619        if not table_name and not column_name:
1620            return False
1621
1622        # Removed
1623        removed = False
1624
1625        # Check if the column already exists in the table
1626        query = f""" SELECT * FROM {table_name} LIMIT 0 """
1627        columns = self.get_query_to_df(query).columns.tolist()
1628        if column_name in columns:
1629            log.debug(f"The {column_name} column exists in the {table_name} table")
1630        else:
1631            log.debug(f"The {column_name} column NOT exists in the {table_name} table")
1632            return False
1633
1634        # Add column in table # ALTER TABLE integers DROP k
1635        add_column_query = f""" ALTER TABLE {table_name} DROP "{column_name}" """
1636        self.execute_query(add_column_query)
1637        removed = True
1638        log.debug(
1639            f"The {column_name} column was successfully dropped to the {table_name} table"
1640        )
1641
1642        return removed

The drop_column function drops a specified column from a given table in a database and returns True if the column was successfully dropped, and False if the column does not exist in the table.

Parameters

column: The column parameter is a dictionary that contains information about the column you want to drop. It has two keys:
table_name: The table_name parameter is the name of the table from which you want to drop a column
column_name: The column_name parameter is the name of the column that you want to drop from the table

Returns

a boolean value. It returns True if the column was successfully dropped from the table, and False if the column does not exist in the table.

def explode_infos( self, prefix: str = None, create_index: bool = False, fields: list = None, force: bool = False, proccess_all_fields_together: bool = False, table: str = None) -> list: View Source

1644    def explode_infos(
1645        self,
1646        prefix: str = None,
1647        create_index: bool = False,
1648        fields: list = None,
1649        force: bool = False,
1650        proccess_all_fields_together: bool = False,
1651        table: str = None,
1652    ) -> list:
1653        """
1654        The `explode_infos` function in Python takes a VCF file and explodes the INFO fields into
1655        individual columns, returning a list of added columns.
1656
1657        :param prefix: The `prefix` parameter is a string that is used as a prefix for the exploded INFO
1658        fields. If the `prefix` is not provided or is set to `None`, the function will use the value of
1659        `self.get_explode_infos_prefix()` as the prefix
1660        :type prefix: str
1661        :param create_index: The `create_index` parameter is a boolean flag that specifies whether to
1662        create indexes on the exploded INFO fields. If set to `True`, indexes will be created; if set to
1663        `False`, indexes will not be created. The default value is `False`, defaults to False
1664        :type create_index: bool (optional)
1665        :param fields: The `fields` parameter in the `explode_infos` function is a list of INFO fields
1666        that you want to explode into individual columns. If this parameter is not provided, all INFO
1667        fields will be exploded. You can specify the INFO fields you want to explode by passing them as
1668        a list to the `
1669        :type fields: list
1670        :param force: The `force` parameter in the `explode_infos` function is a boolean flag that
1671        determines whether to drop and recreate a column if it already exists in the table. If `force`
1672        is set to `True`, the column will be dropped and recreated. If `force` is set to `False,
1673        defaults to False
1674        :type force: bool (optional)
1675        :param proccess_all_fields_together: The `proccess_all_fields_together` parameter is a boolean
1676        flag that determines whether to process all the INFO fields together or individually. If set to
1677        `True`, all the INFO fields will be processed together. If set to `False`, each INFO field will
1678        be processed individually. The default value is, defaults to False
1679        :type proccess_all_fields_together: bool (optional)
1680        :param table: The `table` parameter in the `explode_infos` function is used to specify the name
1681        of the table where the exploded INFO fields will be added as individual columns. If you provide
1682        a value for the `table` parameter, the function will use that table name. If the `table`
1683        parameter is
1684        :type table: str
1685        :return: The `explode_infos` function returns a list of added columns.
1686        """
1687
1688        # drop indexes
1689        self.drop_indexes()
1690
1691        # connexion format
1692        connexion_format = self.get_connexion_format()
1693
1694        # Access
1695        access = self.get_config().get("access", None)
1696
1697        # Added columns
1698        added_columns = []
1699
1700        if access not in ["RO"]:
1701
1702            # prefix
1703            if prefix in [None, True] or not isinstance(prefix, str):
1704                if self.get_explode_infos_prefix() not in [None, True]:
1705                    prefix = self.get_explode_infos_prefix()
1706                else:
1707                    prefix = "INFO/"
1708
1709            # table variants
1710            if table is not None:
1711                table_variants = table
1712            else:
1713                table_variants = self.get_table_variants(clause="select")
1714
1715            # extra infos
1716            try:
1717                extra_infos = self.get_extra_infos()
1718            except:
1719                extra_infos = []
1720
1721            # Header infos
1722            header_infos = self.get_header().infos
1723
1724            log.debug(
1725                f"Explode INFO fields - ADD [{len(header_infos)}] annotations fields"
1726            )
1727
1728            sql_info_alter_table_array = []
1729
1730            # Info fields to check
1731            fields_list = list(header_infos)
1732            if fields:
1733                fields_list += fields
1734            fields_list = set(fields_list)
1735
1736            # If no fields
1737            if not fields:
1738                fields = []
1739
1740            # Translate fields if patterns
1741            fields = self.get_explode_infos_fields(explode_infos_fields=fields)
1742
1743            for info in fields:
1744
1745                info_id_sql = prefix + info
1746
1747                if (
1748                    info in fields_list
1749                    or prefix + info in fields_list
1750                    or info in extra_infos
1751                ):
1752
1753                    log.debug(f"Explode INFO fields - ADD '{info}' annotations fields")
1754
1755                    if info in header_infos:
1756                        info_type = header_infos[info].type
1757                        info_num = header_infos[info].num
1758                    else:
1759                        info_type = "String"
1760                        info_num = 0
1761
1762                    type_sql = self.code_type_map_to_sql.get(info_type, "VARCHAR")
1763                    if info_num != 1:
1764                        type_sql = "VARCHAR"
1765
1766                    # Add field
1767                    added_column = self.add_column(
1768                        table_name=table_variants,
1769                        column_name=info_id_sql,
1770                        column_type=type_sql,
1771                        default_value="null",
1772                        drop=force,
1773                    )
1774
1775                    if added_column:
1776                        added_columns.append(added_column)
1777
1778                    if added_column or force:
1779
1780                        # add field to index
1781                        self.index_additionnal_fields.append(info_id_sql)
1782
1783                        # Update field array
1784                        if connexion_format in ["duckdb"]:
1785                            update_info_field = f"""
1786                            "{info_id_sql}" =
1787                                CASE
1788                                    WHEN REGEXP_EXTRACT(concat(';', INFO), ';{info}=([^;]*)',1) IN ('','.') THEN NULL
1789                                    ELSE REGEXP_EXTRACT(concat(';', INFO), ';{info}=([^;]*)',1)
1790                                END
1791                            """
1792                        elif connexion_format in ["sqlite"]:
1793                            update_info_field = f"""
1794                                "{info_id_sql}" =
1795                                    CASE
1796                                        WHEN instr(INFO, '{info}=') = 0 THEN NULL
1797                                        WHEN instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}),';') = 0 THEN substr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), '=')+1)
1798                                        ELSE substr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), '=')+1, instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}),';')-instr(substr(INFO, instr(INFO, '{info}=')+{len(info)+1}), '=')-1)
1799                                    END
1800                            """
1801
1802                        sql_info_alter_table_array.append(update_info_field)
1803
1804            if sql_info_alter_table_array:
1805
1806                # By chromosomes
1807                try:
1808                    chromosomes_list = list(
1809                        self.get_query_to_df(
1810                            f""" SELECT "#CHROM" FROM {table_variants} GROUP BY "#CHROM" """
1811                        )["#CHROM"]
1812                    )
1813                except:
1814                    chromosomes_list = [None]
1815
1816                for chrom in chromosomes_list:
1817                    log.debug(f"Explode INFO fields - Chromosome {chrom}...")
1818
1819                    # Where clause
1820                    where_clause = ""
1821                    if chrom and len(chromosomes_list) > 1:
1822                        where_clause = f""" WHERE "#CHROM" = '{chrom}' """
1823
1824                    # Update table
1825                    if proccess_all_fields_together:
1826                        sql_info_alter_table_array_join = ", ".join(
1827                            sql_info_alter_table_array
1828                        )
1829                        if sql_info_alter_table_array_join:
1830                            sql_info_alter_table = f"""
1831                                UPDATE {table_variants}
1832                                SET {sql_info_alter_table_array_join}
1833                                {where_clause}
1834                                """
1835                            log.debug(
1836                                f"Explode INFO fields - Explode all {len(sql_info_alter_table_array)} fields..."
1837                            )
1838                            # log.debug(sql_info_alter_table)
1839                            self.conn.execute(sql_info_alter_table)
1840                    else:
1841                        sql_info_alter_num = 0
1842                        for sql_info_alter in sql_info_alter_table_array:
1843                            sql_info_alter_num += 1
1844                            sql_info_alter_table = f"""
1845                                UPDATE {table_variants}
1846                                SET {sql_info_alter}
1847                                {where_clause}
1848                                """
1849                            log.debug(
1850                                f"Explode INFO fields - Explode field {sql_info_alter_num}/{len(sql_info_alter_table_array)}..."
1851                            )
1852                            # log.debug(sql_info_alter_table)
1853                            self.conn.execute(sql_info_alter_table)
1854
1855        # create indexes
1856        if create_index:
1857            self.create_indexes()
1858
1859        return added_columns

The explode_infos function in Python takes a VCF file and explodes the INFO fields into individual columns, returning a list of added columns.

Parameters

prefix: The prefix parameter is a string that is used as a prefix for the exploded INFO fields. If the prefix is not provided or is set to None, the function will use the value of self.get_explode_infos_prefix() as the prefix
create_index: The create_index parameter is a boolean flag that specifies whether to create indexes on the exploded INFO fields. If set to True, indexes will be created; if set to False, indexes will not be created. The default value is False, defaults to False
fields: The fields parameter in the explode_infos function is a list of INFO fields that you want to explode into individual columns. If this parameter is not provided, all INFO fields will be exploded. You can specify the INFO fields you want to explode by passing them as a list to the `
force: The force parameter in the explode_infos function is a boolean flag that determines whether to drop and recreate a column if it already exists in the table. If force is set to True, the column will be dropped and recreated. If force is set to `False, defaults to False
proccess_all_fields_together: The proccess_all_fields_together parameter is a boolean flag that determines whether to process all the INFO fields together or individually. If set to True, all the INFO fields will be processed together. If set to False, each INFO field will be processed individually. The default value is, defaults to False
table: The table parameter in the explode_infos function is used to specify the name of the table where the exploded INFO fields will be added as individual columns. If you provide a value for the table parameter, the function will use that table name. If the table parameter is

Returns

The explode_infos function returns a list of added columns.

def create_indexes(self) -> None: View Source

1861    def create_indexes(self) -> None:
1862        """
1863        Create indexes on the table after insertion
1864        """
1865
1866        # Access
1867        access = self.get_config().get("access", None)
1868
1869        # get table variants
1870        table_variants = self.get_table_variants("FROM")
1871
1872        if self.get_indexing() and access not in ["RO"]:
1873            # Create index
1874            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()} ON {table_variants} ("#CHROM", "POS", "REF", "ALT")'
1875            self.conn.execute(sql_create_table_index)
1876            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_chrom ON {table_variants} ("#CHROM")'
1877            self.conn.execute(sql_create_table_index)
1878            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_pos ON {table_variants} ("POS")'
1879            self.conn.execute(sql_create_table_index)
1880            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_ref ON {table_variants} ( "REF")'
1881            self.conn.execute(sql_create_table_index)
1882            sql_create_table_index = f'CREATE INDEX IF NOT EXISTS idx_{self.get_table_variants()}_alt ON {table_variants} ("ALT")'
1883            self.conn.execute(sql_create_table_index)
1884            for field in self.index_additionnal_fields:
1885                sql_create_table_index = f""" CREATE INDEX IF NOT EXISTS "idx_{self.get_table_variants()}_{field}" ON {table_variants} ("{field}") """
1886                self.conn.execute(sql_create_table_index)

Create indexes on the table after insertion

def drop_indexes(self) -> None: View Source

1888    def drop_indexes(self) -> None:
1889        """
1890        Create indexes on the table after insertion
1891        """
1892
1893        # Access
1894        access = self.get_config().get("access", None)
1895
1896        # get table variants
1897        table_variants = self.get_table_variants("FROM")
1898
1899        # Get database format
1900        connexion_format = self.get_connexion_format()
1901
1902        if access not in ["RO"]:
1903            if connexion_format in ["duckdb"]:
1904                sql_list_indexes = f"SELECT index_name FROM duckdb_indexes WHERE table_name='{table_variants}'"
1905            elif connexion_format in ["sqlite"]:
1906                sql_list_indexes = f"SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='{table_variants}';"
1907
1908            list_indexes = self.conn.execute(sql_list_indexes)
1909            index_names = [row[0] for row in list_indexes.fetchall()]
1910            for index in index_names:
1911                sql_drop_table_index = f""" DROP INDEX IF EXISTS "{index}" """
1912                self.conn.execute(sql_drop_table_index)

Create indexes on the table after insertion

def read_vcf_header(self, f) -> list: View Source

1914    def read_vcf_header(self, f) -> list:
1915        """
1916        It reads the header of a VCF file and returns a list of the header lines
1917
1918        :param f: the file object
1919        :return: The header lines of the VCF file.
1920        """
1921
1922        header_list = []
1923        for line in f:
1924            header_list.append(line)
1925            if line.startswith("#CHROM"):
1926                break
1927        return header_list

It reads the header of a VCF file and returns a list of the header lines

Parameters

f: the file object

Returns

The header lines of the VCF file.

def read_vcf_header_file(self, file: str = None) -> list: View Source

1929    def read_vcf_header_file(self, file: str = None) -> list:
1930        """
1931        The `read_vcf_header_file` function reads the header of a VCF file, handling both compressed and
1932        uncompressed files.
1933
1934        :param file: The `file` parameter is a string that represents the path to the VCF header file
1935        that you want to read. It is an optional parameter, so if you don't provide a value, it will
1936        default to `None`
1937        :type file: str
1938        :return: The function `read_vcf_header_file` returns a list.
1939        """
1940
1941        if self.get_input_compressed(input_file=file):
1942            with bgzf.open(file, "rt") as f:
1943                return self.read_vcf_header(f=f)
1944        else:
1945            with open(file, "rt") as f:
1946                return self.read_vcf_header(f=f)

The read_vcf_header_file function reads the header of a VCF file, handling both compressed and uncompressed files.

Parameters

file: The file parameter is a string that represents the path to the VCF header file that you want to read. It is an optional parameter, so if you don't provide a value, it will default to None

Returns

The function read_vcf_header_file returns a list.

def execute_query(self, query: str): View Source

1948    def execute_query(self, query: str):
1949        """
1950        It takes a query as an argument, executes it, and returns the results
1951
1952        :param query: The query to be executed
1953        :return: The result of the query is being returned.
1954        """
1955        if query:
1956            return self.conn.execute(query)  # .fetchall()
1957        else:
1958            return None

It takes a query as an argument, executes it, and returns the results

Parameters

query: The query to be executed

Returns

The result of the query is being returned.

1960    def export_output(
1961        self,
1962        output_file: str | None = None,
1963        output_header: str | None = None,
1964        export_header: bool = True,
1965        query: str | None = None,
1966        parquet_partitions: list | None = None,
1967        chunk_size: int | None = None,
1968        threads: int | None = None,
1969        sort: bool = False,
1970        index: bool = False,
1971        order_by: str | None = None,
1972    ) -> bool:
1973        """
1974        The `export_output` function exports data from a VCF file to a specified output file in various
1975        formats, including VCF, CSV, TSV, PSV, and Parquet.
1976
1977        :param output_file: The `output_file` parameter is a string that specifies the name of the
1978        output file to be generated by the function. This is where the exported data will be saved
1979        :type output_file: str
1980        :param output_header: The `output_header` parameter is a string that specifies the name of the
1981        file where the header of the VCF file will be exported. If this parameter is not provided, the
1982        header will be exported to a file with the same name as the `output_file` parameter, but with
1983        the extension "
1984        :type output_header: str
1985        :param export_header: The `export_header` parameter is a boolean flag that determines whether
1986        the header of a VCF file should be exported to a separate file or not. If `export_header` is
1987        True, the header will be exported to a file. If `export_header` is False, the header will not
1988        be, defaults to True, if output format is not VCF
1989        :type export_header: bool (optional)
1990        :param query: The `query` parameter is an optional SQL query that can be used to filter and
1991        select specific data from the VCF file before exporting it. If provided, only the data that
1992        matches the query will be exported
1993        :type query: str
1994        :param parquet_partitions: The `parquet_partitions` parameter is a list that specifies the
1995        columns to be used for partitioning the Parquet file during export. Partitioning is a way to
1996        organize data in a hierarchical directory structure based on the values of one or more columns.
1997        This can improve query performance when working with large datasets
1998        :type parquet_partitions: list
1999        :param chunk_size: The `chunk_size` parameter specifies the number of
2000        records in batch when exporting data in Parquet format. This parameter is used for
2001        partitioning the Parquet file into multiple files.
2002        :type chunk_size: int
2003        :param threads: The `threads` parameter is an optional parameter that specifies the number of
2004        threads to be used during the export process. It determines the level of parallelism and can
2005        improve the performance of the export operation. If not provided, the function will use the
2006        default number of threads
2007        :type threads: int
2008        :param sort: The `sort` parameter is a boolean flag that determines whether the output file
2009        should be sorted or not. If `sort` is set to `True`, the output file will be sorted based on the
2010        genomic coordinates of the variants. By default, the value of `sort` is `False`, defaults to
2011        False
2012        :type sort: bool (optional)
2013        :param index: The `index` parameter is a boolean flag that determines whether an index should be
2014        created on the output file. If `index` is True, an index will be created. If `index` is False,
2015        no index will be created. The default value is False, defaults to False
2016        :type index: bool (optional)
2017        :param order_by: The `order_by` parameter is a string that specifies the column(s) to use for
2018        sorting the output file. This parameter is only applicable when exporting data in VCF format
2019        :type order_by: str
2020        :return: a boolean value. It checks if the output file exists and returns True if it does, or
2021        None if it doesn't.
2022        """
2023
2024        # Log
2025        log.info("Exporting...")
2026
2027        # Full path
2028        output_file = full_path(output_file)
2029        output_header = full_path(output_header)
2030
2031        # Config
2032        config = self.get_config()
2033
2034        # Param
2035        param = self.get_param()
2036
2037        # Tmp files to remove
2038        tmp_to_remove = []
2039
2040        # If no output, get it
2041        if not output_file:
2042            output_file = self.get_output()
2043
2044        # If not threads
2045        if not threads:
2046            threads = self.get_threads()
2047
2048        # Auto header name with extension
2049        if export_header or output_header:
2050            if not output_header:
2051                output_header = f"{output_file}.hdr"
2052            # Export header
2053            self.export_header(output_file=output_file)
2054
2055        # Switch off export header if VCF output
2056        output_file_type = get_file_format(output_file)
2057        if output_file_type in ["vcf"]:
2058            export_header = False
2059            tmp_to_remove.append(output_header)
2060
2061        # Chunk size
2062        if not chunk_size:
2063            chunk_size = config.get("chunk_size", None)
2064
2065        # Parquet partition
2066        if not parquet_partitions:
2067            parquet_partitions = param.get("export", {}).get("parquet_partitions", None)
2068        if parquet_partitions and isinstance(parquet_partitions, str):
2069            parquet_partitions = parquet_partitions.split(",")
2070
2071        # Order by
2072        if not order_by:
2073            order_by = param.get("export", {}).get("order_by", "")
2074
2075        # Header in output
2076        header_in_output = param.get("export", {}).get("include_header", False)
2077
2078        # Database
2079        database_source = self.get_connexion()
2080
2081        # Connexion format
2082        connexion_format = self.get_connexion_format()
2083
2084        # Explode infos
2085        if self.get_explode_infos():
2086            self.explode_infos(
2087                prefix=self.get_explode_infos_prefix(),
2088                fields=self.get_explode_infos_fields(),
2089                force=False,
2090            )
2091
2092        # if connexion_format in ["sqlite"] or query:
2093        if connexion_format in ["sqlite"]:
2094
2095            # Export in Parquet
2096            random_tmp = "".join(
2097                random.choice(string.ascii_lowercase) for i in range(10)
2098            )
2099            database_source = f"""{output_file}.{random_tmp}.database_export.parquet"""
2100            tmp_to_remove.append(database_source)
2101
2102            # Table Variants
2103            table_variants = self.get_table_variants()
2104
2105            # Create export query
2106            sql_query_export_subquery = f"""
2107                SELECT * FROM {table_variants}
2108                """
2109
2110            # Write source file
2111            fp.write(database_source, self.get_query_to_df(sql_query_export_subquery))
2112
2113        # Create database
2114        database = Database(
2115            database=database_source,
2116            table="variants",
2117            header_file=output_header,
2118            conn_config=self.get_connexion_config(),
2119        )
2120
2121        # Existing colomns header
2122        # existing_columns_header = database.get_header_file_columns(output_header)
2123        existing_columns_header = database.get_header_columns_from_database()
2124
2125        # Export file
2126        database.export(
2127            output_database=output_file,
2128            output_header=output_header,
2129            existing_columns_header=existing_columns_header,
2130            parquet_partitions=parquet_partitions,
2131            chunk_size=chunk_size,
2132            threads=threads,
2133            sort=sort,
2134            index=index,
2135            header_in_output=header_in_output,
2136            order_by=order_by,
2137            query=query,
2138            export_header=export_header,
2139        )
2140
2141        # Remove
2142        remove_if_exists(tmp_to_remove)
2143
2144        return (os.path.exists(output_file) or None) and (
2145            os.path.exists(output_file) or None
2146        )

The export_output function exports data from a VCF file to a specified output file in various formats, including VCF, CSV, TSV, PSV, and Parquet.

Parameters

output_file: The output_file parameter is a string that specifies the name of the output file to be generated by the function. This is where the exported data will be saved
output_header: The output_header parameter is a string that specifies the name of the file where the header of the VCF file will be exported. If this parameter is not provided, the header will be exported to a file with the same name as the output_file parameter, but with the extension "
export_header: The export_header parameter is a boolean flag that determines whether the header of a VCF file should be exported to a separate file or not. If export_header is True, the header will be exported to a file. If export_header is False, the header will not be, defaults to True, if output format is not VCF
query: The query parameter is an optional SQL query that can be used to filter and select specific data from the VCF file before exporting it. If provided, only the data that matches the query will be exported
parquet_partitions: The parquet_partitions parameter is a list that specifies the columns to be used for partitioning the Parquet file during export. Partitioning is a way to organize data in a hierarchical directory structure based on the values of one or more columns. This can improve query performance when working with large datasets
chunk_size: The chunk_size parameter specifies the number of records in batch when exporting data in Parquet format. This parameter is used for partitioning the Parquet file into multiple files.
threads: The threads parameter is an optional parameter that specifies the number of threads to be used during the export process. It determines the level of parallelism and can improve the performance of the export operation. If not provided, the function will use the default number of threads
sort: The sort parameter is a boolean flag that determines whether the output file should be sorted or not. If sort is set to True, the output file will be sorted based on the genomic coordinates of the variants. By default, the value of sort is False, defaults to False
index: The index parameter is a boolean flag that determines whether an index should be created on the output file. If index is True, an index will be created. If index is False, no index will be created. The default value is False, defaults to False
order_by: The order_by parameter is a string that specifies the column(s) to use for sorting the output file. This parameter is only applicable when exporting data in VCF format

Returns

a boolean value. It checks if the output file exists and returns True if it does, or None if it doesn't.

def get_extra_infos(self, table: str = None) -> list: View Source

2148    def get_extra_infos(self, table: str = None) -> list:
2149        """
2150        The `get_extra_infos` function returns a list of columns that are in a specified table but not
2151        in the header.
2152
2153        :param table: The `table` parameter in the `get_extra_infos` function is used to specify the
2154        name of the table from which you want to retrieve the extra columns that are not present in the
2155        header. If the `table` parameter is not provided when calling the function, it will default to
2156        using the variants
2157        :type table: str
2158        :return: A list of columns that are in the specified table but not in the header of the table.
2159        """
2160
2161        header_columns = []
2162
2163        if not table:
2164            table = self.get_table_variants(clause="from")
2165            header_columns = self.get_header_columns()
2166
2167        # Check all columns in the database
2168        query = f""" SELECT * FROM {table} LIMIT 1 """
2169        log.debug(f"query {query}")
2170        table_columns = self.get_query_to_df(query).columns.tolist()
2171        extra_columns = []
2172
2173        # Construct extra infos (not in header)
2174        for column in table_columns:
2175            if column not in header_columns:
2176                extra_columns.append(column)
2177
2178        return extra_columns

The get_extra_infos function returns a list of columns that are in a specified table but not in the header.

Parameters

table: The table parameter in the get_extra_infos function is used to specify the name of the table from which you want to retrieve the extra columns that are not present in the header. If the table parameter is not provided when calling the function, it will default to using the variants

Returns

A list of columns that are in the specified table but not in the header of the table.

def get_extra_infos_sql(self, table: str = None) -> str: View Source

2180    def get_extra_infos_sql(self, table: str = None) -> str:
2181        """
2182        It returns a string of the extra infos, separated by commas, and each extra info is surrounded
2183        by double quotes
2184
2185        :param table: The name of the table to get the extra infos from. If None, the default table is
2186        used
2187        :type table: str
2188        :return: A string of the extra infos
2189        """
2190
2191        return ", ".join(
2192            ['"' + str(elem) + '"' for elem in self.get_extra_infos(table=table)]
2193        )

It returns a string of the extra infos, separated by commas, and each extra info is surrounded by double quotes

Parameters

table: The name of the table to get the extra infos from. If None, the default table is used

Returns

A string of the extra infos

def export_header( self, header_name: str = None, output_file: str = None, output_file_ext: str = '.hdr', clean_header: bool = True, remove_chrom_line: bool = False) -> str: View Source

2195    def export_header(
2196        self,
2197        header_name: str = None,
2198        output_file: str = None,
2199        output_file_ext: str = ".hdr",
2200        clean_header: bool = True,
2201        remove_chrom_line: bool = False,
2202    ) -> str:
2203        """
2204        The `export_header` function takes a VCF file, extracts the header, modifies it according to
2205        specified options, and writes it to a new file.
2206
2207        :param header_name: The `header_name` parameter is the name of the header file to be created. If
2208        this parameter is not specified, the header will be written to the output file
2209        :type header_name: str
2210        :param output_file: The `output_file` parameter in the `export_header` function is used to
2211        specify the name of the output file where the header will be written. If this parameter is not
2212        provided, the header will be written to a temporary file
2213        :type output_file: str
2214        :param output_file_ext: The `output_file_ext` parameter in the `export_header` function is a
2215        string that represents the extension of the output header file. By default, it is set to ".hdr"
2216        if not specified by the user. This extension will be appended to the `output_file` name to
2217        create the final, defaults to .hdr
2218        :type output_file_ext: str (optional)
2219        :param clean_header: The `clean_header` parameter in the `export_header` function is a boolean
2220        flag that determines whether the header should be cleaned or not. When `clean_header` is set to
2221        `True`, the function will clean the header by modifying certain lines based on a specific
2222        pattern. If `clean_header`, defaults to True
2223        :type clean_header: bool (optional)
2224        :param remove_chrom_line: The `remove_chrom_line` parameter in the `export_header` function is a
2225        boolean flag that determines whether the #CHROM line should be removed from the header before
2226        writing it to the output file. If set to `True`, the #CHROM line will be removed; if set to `,
2227        defaults to False
2228        :type remove_chrom_line: bool (optional)
2229        :return: The function `export_header` returns the name of the temporary header file that is
2230        created.
2231        """
2232
2233        if not header_name and not output_file:
2234            output_file = self.get_output()
2235
2236        if self.get_header():
2237
2238            # Get header object
2239            header_obj = self.get_header()
2240
2241            # Create database
2242            db_for_header = Database(database=self.get_input())
2243
2244            # Get real columns in the file
2245            db_header_columns = db_for_header.get_columns()
2246
2247            with tempfile.TemporaryDirectory() as tmpdir:
2248
2249                # Write header file
2250                header_file_tmp = os.path.join(tmpdir, "header")
2251                f = open(header_file_tmp, "w")
2252                vcf.Writer(f, header_obj)
2253                f.close()
2254
2255                # Replace #CHROM line with rel columns
2256                header_list = db_for_header.read_header_file(
2257                    header_file=header_file_tmp
2258                )
2259                header_list[-1] = "\t".join(db_header_columns)
2260
2261                # Remove CHROM line
2262                if remove_chrom_line:
2263                    header_list.pop()
2264
2265                # Clean header
2266                if clean_header:
2267                    header_list_clean = []
2268                    for head in header_list:
2269                        # Clean head for malformed header
2270                        head_clean = head
2271                        head_clean = re.subn(
2272                            "##FORMAT=<ID=(.*),Number=(.*),Type=Flag",
2273                            r"##FORMAT=<ID=\1,Number=\2,Type=String",
2274                            head_clean,
2275                            2,
2276                        )[0]
2277                        # Write header
2278                        header_list_clean.append(head_clean)
2279                    header_list = header_list_clean
2280
2281            tmp_header_name = output_file + output_file_ext
2282
2283            f = open(tmp_header_name, "w")
2284            for line in header_list:
2285                f.write(line)
2286            f.close()
2287
2288        return tmp_header_name

The export_header function takes a VCF file, extracts the header, modifies it according to specified options, and writes it to a new file.

Parameters

header_name: The header_name parameter is the name of the header file to be created. If this parameter is not specified, the header will be written to the output file
output_file: The output_file parameter in the export_header function is used to specify the name of the output file where the header will be written. If this parameter is not provided, the header will be written to a temporary file
output_file_ext: The output_file_ext parameter in the export_header function is a string that represents the extension of the output header file. By default, it is set to ".hdr" if not specified by the user. This extension will be appended to the output_file name to create the final, defaults to .hdr
clean_header: The clean_header parameter in the export_header function is a boolean flag that determines whether the header should be cleaned or not. When clean_header is set to True, the function will clean the header by modifying certain lines based on a specific pattern. If clean_header, defaults to True
remove_chrom_line: The remove_chrom_line parameter in the export_header function is a boolean flag that determines whether the #CHROM line should be removed from the header before writing it to the output file. If set to True, the #CHROM line will be removed; if set to `, defaults to False

Returns

The function export_header returns the name of the temporary header file that is created.

def export_variant_vcf( self, vcf_file, remove_info: bool = False, add_samples: bool = True, list_samples: list = [], where_clause: str = '', index: bool = False, threads: int | None = None) -> bool | None: View Source

2290    def export_variant_vcf(
2291        self,
2292        vcf_file,
2293        remove_info: bool = False,
2294        add_samples: bool = True,
2295        list_samples: list = [],
2296        where_clause: str = "",
2297        index: bool = False,
2298        threads: int | None = None,
2299    ) -> bool | None:
2300        """
2301        The `export_variant_vcf` function exports a VCF file with specified samples, allowing options to
2302        remove INFO field, add samples, and control compression and indexing.
2303
2304        :param vcf_file: The `vcf_file` parameter is the name of the file where the VCF data will be
2305        written to. It is the output file that will contain the filtered VCF data based on the specified
2306        parameters
2307        :param remove_info: The `remove_info` parameter in the `export_variant_vcf` function is a
2308        boolean flag that determines whether to remove the INFO field from the output VCF file. If set
2309        to `True`, the INFO field will be removed. If set to `False`, the INFO field will be included
2310        in, defaults to False
2311        :type remove_info: bool (optional)
2312        :param add_samples: The `add_samples` parameter is a boolean parameter that determines whether
2313        the samples should be added to the VCF file or not. If set to True, the samples will be added.
2314        If set to False, the samples will be removed. The default value is True, defaults to True
2315        :type add_samples: bool (optional)
2316        :param list_samples: The `list_samples` parameter is a list of samples that you want to include
2317        in the output VCF file. By default, all samples will be included. If you provide a list of
2318        samples, only those samples will be included in the output file
2319        :type list_samples: list
2320        :param index: The `index` parameter in the `export_variant_vcf` function is a boolean flag that
2321        determines whether or not to create an index for the output VCF file. If `index` is set to
2322        `True`, the output VCF file will be indexed using tabix. If `index`, defaults to False
2323        :type index: bool (optional)
2324        :param threads: The `threads` parameter in the `export_variant_vcf` function specifies the
2325        number of threads to use for exporting the VCF file. It determines how many parallel threads
2326        will be used during the export process. More threads can potentially speed up the export process
2327        by utilizing multiple cores of the processor. If
2328        :type threads: int | None
2329        :return: The `export_variant_vcf` function returns the result of calling the `export_output`
2330        method with various parameters including the output file, query, threads, sort flag, and index
2331        flag. The `export_output` method is responsible for exporting the VCF data based on the
2332        specified parameters and configurations provided in the `export_variant_vcf` function.
2333        """
2334
2335        # Config
2336        config = self.get_config()
2337
2338        # Extract VCF
2339        log.debug("Export VCF...")
2340
2341        # Table variants
2342        table_variants = self.get_table_variants()
2343
2344        # Threads
2345        if not threads:
2346            threads = self.get_threads()
2347
2348        # Info fields
2349        if remove_info:
2350            if not isinstance(remove_info, str):
2351                remove_info = "."
2352            info_field = f"""'{remove_info}' as INFO"""
2353        else:
2354            info_field = "INFO"
2355
2356        # Samples fields
2357        if add_samples:
2358            if not list_samples:
2359                list_samples = self.get_header_sample_list()
2360            if list_samples:
2361                samples_fields = " , FORMAT , " + " , ".join(list_samples)
2362            else:
2363                samples_fields = ""
2364            log.debug(f"samples_fields: {samples_fields}")
2365        else:
2366            samples_fields = ""
2367
2368        # Where clause
2369        if where_clause is None:
2370            where_clause = ""
2371
2372        # Variants
2373        select_fields = """ "#CHROM", POS, ID, REF, ALT, QUAL, FILTER """
2374        sql_query_select = f""" SELECT {select_fields}, {info_field} {samples_fields} FROM {table_variants} {where_clause} """
2375        log.debug(f"sql_query_select={sql_query_select}")
2376
2377        return self.export_output(
2378            output_file=vcf_file,
2379            output_header=None,
2380            export_header=True,
2381            query=sql_query_select,
2382            parquet_partitions=None,
2383            chunk_size=config.get("chunk_size", None),
2384            threads=threads,
2385            sort=True,
2386            index=index,
2387            order_by=None,
2388        )

The export_variant_vcf function exports a VCF file with specified samples, allowing options to remove INFO field, add samples, and control compression and indexing.

Parameters

vcf_file: The vcf_file parameter is the name of the file where the VCF data will be written to. It is the output file that will contain the filtered VCF data based on the specified parameters
remove_info: The remove_info parameter in the export_variant_vcf function is a boolean flag that determines whether to remove the INFO field from the output VCF file. If set to True, the INFO field will be removed. If set to False, the INFO field will be included in, defaults to False
add_samples: The add_samples parameter is a boolean parameter that determines whether the samples should be added to the VCF file or not. If set to True, the samples will be added. If set to False, the samples will be removed. The default value is True, defaults to True
list_samples: The list_samples parameter is a list of samples that you want to include in the output VCF file. By default, all samples will be included. If you provide a list of samples, only those samples will be included in the output file
index: The index parameter in the export_variant_vcf function is a boolean flag that determines whether or not to create an index for the output VCF file. If index is set to True, the output VCF file will be indexed using tabix. If index, defaults to False
threads: The threads parameter in the export_variant_vcf function specifies the number of threads to use for exporting the VCF file. It determines how many parallel threads will be used during the export process. More threads can potentially speed up the export process by utilizing multiple cores of the processor. If

Returns

The export_variant_vcf function returns the result of calling the export_output method with various parameters including the output file, query, threads, sort flag, and index flag. The export_output method is responsible for exporting the VCF data based on the specified parameters and configurations provided in the export_variant_vcf function.

def run_commands(self, commands: list = [], threads: int = 1) -> None: View Source

2390    def run_commands(self, commands: list = [], threads: int = 1) -> None:
2391        """
2392        It takes a list of commands and runs them in parallel using the number of threads specified
2393
2394        :param commands: A list of commands to run
2395        :param threads: The number of threads to use, defaults to 1 (optional)
2396        """
2397
2398        run_parallel_commands(commands, threads)

It takes a list of commands and runs them in parallel using the number of threads specified

Parameters

commands: A list of commands to run
threads: The number of threads to use, defaults to 1 (optional)

def get_threads(self, default: int = 1) -> int: View Source

2400    def get_threads(self, default: int = 1) -> int:
2401        """
2402        This function returns the number of threads to use for a job, with a default value of 1 if not
2403        specified.
2404
2405        :param default: The `default` parameter in the `get_threads` method is used to specify the
2406        default number of threads to use if no specific value is provided. If no value is provided for
2407        the `threads` parameter in the configuration or input parameters, the `default` value will be
2408        used, defaults to 1
2409        :type default: int (optional)
2410        :return: the number of threads to use for the current job.
2411        """
2412
2413        # Config
2414        config = self.get_config()
2415
2416        # Param
2417        param = self.get_param()
2418
2419        # Input threads
2420        input_thread = param.get("threads", config.get("threads", None))
2421
2422        # Check threads
2423        if not input_thread:
2424            threads = default
2425        elif int(input_thread) <= 0:
2426            threads = os.cpu_count()
2427        else:
2428            threads = int(input_thread)
2429        return threads

This function returns the number of threads to use for a job, with a default value of 1 if not specified.

Parameters

default: The default parameter in the get_threads method is used to specify the default number of threads to use if no specific value is provided. If no value is provided for the threads parameter in the configuration or input parameters, the default value will be used, defaults to 1

Returns

the number of threads to use for the current job.

def get_memory(self, default: str = None) -> str: View Source

2431    def get_memory(self, default: str = None) -> str:
2432        """
2433        This function retrieves the memory value from parameters or configuration with a default value
2434        if not found.
2435
2436        :param default: The `get_memory` function takes in a default value as a string parameter. This
2437        default value is used as a fallback in case the `memory` parameter is not provided in the
2438        `param` dictionary or the `config` dictionary. If `memory` is not found in either dictionary,
2439        the function
2440        :type default: str
2441        :return: The `get_memory` function returns a string value representing the memory parameter. If
2442        the `input_memory` is provided in the parameters, it will return that value. Otherwise, it will
2443        return the default value provided as an argument to the function.
2444        """
2445
2446        # Config
2447        config = self.get_config()
2448
2449        # Param
2450        param = self.get_param()
2451
2452        # Input threads
2453        input_memory = param.get("memory", config.get("memory", None))
2454
2455        # Check threads
2456        if input_memory:
2457            memory = input_memory
2458        else:
2459            memory = default
2460
2461        return memory

This function retrieves the memory value from parameters or configuration with a default value if not found.

Parameters

default: The get_memory function takes in a default value as a string parameter. This default value is used as a fallback in case the memory parameter is not provided in the param dictionary or the config dictionary. If memory is not found in either dictionary, the function

Returns

The get_memory function returns a string value representing the memory parameter. If the input_memory is provided in the parameters, it will return that value. Otherwise, it will return the default value provided as an argument to the function.

def update_from_vcf(self, vcf_file: str) -> None: View Source

2463    def update_from_vcf(self, vcf_file: str) -> None:
2464        """
2465        > If the database is duckdb, then use the parquet method, otherwise use the sqlite method
2466
2467        :param vcf_file: the path to the VCF file
2468        """
2469
2470        connexion_format = self.get_connexion_format()
2471
2472        if connexion_format in ["duckdb"]:
2473            self.update_from_vcf_duckdb(vcf_file)
2474        elif connexion_format in ["sqlite"]:
2475            self.update_from_vcf_sqlite(vcf_file)

If the database is duckdb, then use the parquet method, otherwise use the sqlite method

Parameters

vcf_file: the path to the VCF file

def update_from_vcf_duckdb(self, vcf_file: str) -> None: View Source

2477    def update_from_vcf_duckdb(self, vcf_file: str) -> None:
2478        """
2479        It takes a VCF file and updates the INFO column of the variants table in the database with the
2480        INFO column of the VCF file
2481
2482        :param vcf_file: the path to the VCF file
2483        """
2484
2485        # varaints table
2486        table_variants = self.get_table_variants()
2487
2488        # Loading VCF into temporaire table
2489        skip = self.get_header_length(file=vcf_file)
2490        vcf_df = pd.read_csv(
2491            vcf_file,
2492            sep="\t",
2493            engine="c",
2494            skiprows=skip,
2495            header=0,
2496            low_memory=False,
2497        )
2498        sql_query_update = f"""
2499        UPDATE {table_variants} as table_variants
2500            SET INFO = concat(
2501                            CASE
2502                                WHEN INFO NOT IN ('', '.')
2503                                THEN INFO
2504                                ELSE ''
2505                            END,
2506                            (
2507                                SELECT 
2508                                    concat(
2509                                        CASE
2510                                            WHEN table_variants.INFO NOT IN ('','.') AND table_parquet.INFO NOT IN ('','.')
2511                                            THEN ';'
2512                                            ELSE ''
2513                                        END
2514                                        ,
2515                                        CASE
2516                                            WHEN table_parquet.INFO NOT IN ('','.')
2517                                            THEN table_parquet.INFO
2518                                            ELSE ''
2519                                        END
2520                                    )
2521                                FROM vcf_df as table_parquet
2522                                        WHERE CAST(table_parquet.\"#CHROM\" AS VARCHAR) = CAST(table_variants.\"#CHROM\" AS VARCHAR)
2523                                        AND table_parquet.\"POS\" = table_variants.\"POS\"
2524                                        AND table_parquet.\"ALT\" = table_variants.\"ALT\"
2525                                        AND table_parquet.\"REF\" = table_variants.\"REF\"
2526                                        AND table_parquet.INFO NOT IN ('','.')
2527                            )
2528                        )
2529            ;
2530            """
2531        self.conn.execute(sql_query_update)

It takes a VCF file and updates the INFO column of the variants table in the database with the INFO column of the VCF file

Parameters

vcf_file: the path to the VCF file

def update_from_vcf_sqlite(self, vcf_file: str) -> None: View Source

2533    def update_from_vcf_sqlite(self, vcf_file: str) -> None:
2534        """
2535        It creates a temporary table in the SQLite database, loads the VCF file into the temporary
2536        table, then updates the INFO column of the variants table with the INFO column of the temporary
2537        table
2538
2539        :param vcf_file: The path to the VCF file you want to update the database with
2540        """
2541
2542        # Create a temporary table for the VCF
2543        table_vcf = "tmp_vcf"
2544        sql_create = (
2545            f"CREATE TEMPORARY TABLE {table_vcf} AS SELECT * FROM variants WHERE 0"
2546        )
2547        self.conn.execute(sql_create)
2548
2549        # Loading VCF into temporaire table
2550        vcf_df = pd.read_csv(
2551            vcf_file, sep="\t", comment="#", header=None, low_memory=False
2552        )
2553        vcf_df.columns = ["#CHROM", "POS", "ID", "REF", "ALT", "QUAL", "FILTER", "INFO"]
2554        vcf_df.to_sql(table_vcf, self.conn, if_exists="append", index=False)
2555
2556        # Update table 'variants' with VCF data
2557        # warning: CONCAT as || operator
2558        sql_query_update = f"""
2559            UPDATE variants as table_variants
2560            SET INFO = CASE
2561                            WHEN INFO NOT IN ('', '.')
2562                            THEN INFO
2563                            ELSE ''
2564                        END ||
2565                        (
2566                        SELECT 
2567                            CASE 
2568                                WHEN table_variants.INFO NOT IN ('','.') 
2569                                    AND table_vcf.INFO NOT IN ('','.')  
2570                                THEN ';' 
2571                                ELSE '' 
2572                            END || 
2573                            CASE 
2574                                WHEN table_vcf.INFO NOT IN ('','.') 
2575                                THEN table_vcf.INFO 
2576                                ELSE '' 
2577                            END
2578                        FROM {table_vcf} as table_vcf
2579                        WHERE table_vcf.\"#CHROM\" = table_variants.\"#CHROM\"
2580                            AND table_vcf.\"POS\" = table_variants.\"POS\"
2581                            AND table_vcf.\"ALT\" = table_variants.\"ALT\"
2582                            AND table_vcf.\"REF\" = table_variants.\"REF\"
2583                        )
2584        """
2585        self.conn.execute(sql_query_update)
2586
2587        # Drop temporary table
2588        sql_drop = f"DROP TABLE {table_vcf}"
2589        self.conn.execute(sql_drop)

It creates a temporary table in the SQLite database, loads the VCF file into the temporary table, then updates the INFO column of the variants table with the INFO column of the temporary table

Parameters

vcf_file: The path to the VCF file you want to update the database with

def drop_variants_table(self) -> None: View Source

2591    def drop_variants_table(self) -> None:
2592        """
2593        > This function drops the variants table
2594        """
2595
2596        table_variants = self.get_table_variants()
2597        sql_table_variants = f"DROP TABLE IF EXISTS {table_variants}"
2598        self.conn.execute(sql_table_variants)

This function drops the variants table

def set_variant_id(self, variant_id_column: str = 'variant_id', force: bool = None) -> str: View Source

2600    def set_variant_id(
2601        self, variant_id_column: str = "variant_id", force: bool = None
2602    ) -> str:
2603        """
2604        It adds a column to the variants table called `variant_id` and populates it with a hash of the
2605        `#CHROM`, `POS`, `REF`, and `ALT` columns
2606
2607        :param variant_id_column: The name of the column to be created in the variants table, defaults
2608        to variant_id
2609        :type variant_id_column: str (optional)
2610        :param force: If True, the variant_id column will be created even if it already exists
2611        :type force: bool
2612        :return: The name of the column that contains the variant_id
2613        """
2614
2615        # Assembly
2616        assembly = self.get_param().get(
2617            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
2618        )
2619
2620        # INFO/Tag prefix
2621        prefix = self.get_explode_infos_prefix()
2622
2623        # Explode INFO/SVTYPE
2624        added_columns = self.explode_infos(prefix=prefix, fields=["SVTYPE"])
2625
2626        # variants table
2627        table_variants = self.get_table_variants()
2628
2629        # variant_id column
2630        if not variant_id_column:
2631            variant_id_column = "variant_id"
2632
2633        # Creta variant_id column
2634        if "variant_id" not in self.get_extra_infos() or force:
2635
2636            # Create column
2637            self.add_column(
2638                table_name=table_variants,
2639                column_name=variant_id_column,
2640                column_type="UBIGINT",
2641                default_value="0",
2642            )
2643
2644            # Update column
2645            self.conn.execute(
2646                f"""
2647                    UPDATE {table_variants}
2648                    SET "{variant_id_column}" = hash('{assembly}', "#CHROM", "POS", "REF", "ALT", '"{prefix}SVTYPE"')
2649                """
2650            )
2651
2652        # Remove added columns
2653        for added_column in added_columns:
2654            self.drop_column(column=added_column)
2655
2656        # return variant_id column name
2657        return variant_id_column

It adds a column to the variants table called variant_id and populates it with a hash of the #CHROM, POS, REF, and ALT columns

Parameters

variant_id_column: The name of the column to be created in the variants table, defaults to variant_id
force: If True, the variant_id column will be created even if it already exists

Returns

The name of the column that contains the variant_id

def get_variant_id_column(self, variant_id_column: str = 'variant_id', force: bool = None) -> str: View Source

2659    def get_variant_id_column(
2660        self, variant_id_column: str = "variant_id", force: bool = None
2661    ) -> str:
2662        """
2663        This function returns the variant_id column name
2664
2665        :param variant_id_column: The name of the column in the dataframe that contains the variant IDs,
2666        defaults to variant_id
2667        :type variant_id_column: str (optional)
2668        :param force: If True, will force the variant_id to be set to the value of variant_id_column. If
2669        False, will only set the variant_id if it is not already set. If None, will set the variant_id
2670        if it is not already set, or if it is set
2671        :type force: bool
2672        :return: The variant_id column name.
2673        """
2674
2675        return self.set_variant_id(variant_id_column=variant_id_column, force=force)

This function returns the variant_id column name

Parameters

variant_id_column: The name of the column in the dataframe that contains the variant IDs, defaults to variant_id
force: If True, will force the variant_id to be set to the value of variant_id_column. If False, will only set the variant_id if it is not already set. If None, will set the variant_id if it is not already set, or if it is set

Returns

The variant_id column name.

def scan_databases( self, database_formats: list = ['parquet'], database_releases: list = ['current']) -> dict: View Source

2681    def scan_databases(
2682        self,
2683        database_formats: list = ["parquet"],
2684        database_releases: list = ["current"],
2685    ) -> dict:
2686        """
2687        The function `scan_databases` scans for available databases based on specified formats and
2688        releases.
2689
2690        :param database_formats: The `database_formats` parameter is a list that specifies the formats
2691        of the databases to be scanned. In this case, the accepted format is "parquet"
2692        :type database_formats: list ["parquet"]
2693        :param database_releases: The `database_releases` parameter is a list that specifies the
2694        releases of the databases to be scanned. In the provided function, the default value for
2695        `database_releases` is set to `["current"]`, meaning that by default, the function will scan
2696        databases that are in the "current"
2697        :type database_releases: list
2698        :return: The function `scan_databases` returns a dictionary containing information about
2699        databases that match the specified formats and releases.
2700        """
2701
2702        # Config
2703        config = self.get_config()
2704
2705        # Param
2706        param = self.get_param()
2707
2708        # Param - Assembly
2709        assembly = param.get("assembly", config.get("assembly", None))
2710        if not assembly:
2711            assembly = DEFAULT_ASSEMBLY
2712            log.warning(f"Default assembly '{assembly}'")
2713
2714        # Scan for availabled databases
2715        log.info(
2716            f"Annotations - Check annotation parameters - Scan existing databases - Assembly {[assembly]} - Formats {database_formats} - Releases {database_releases}..."
2717        )
2718        databases_infos_dict = databases_infos(
2719            database_folder_releases=database_releases,
2720            database_formats=database_formats,
2721            assembly=assembly,
2722            config=config,
2723        )
2724        log.info(
2725            f"Annotations - Check annotation parameters - Scan existing databases - {len(databases_infos_dict)} databases found"
2726        )
2727
2728        return databases_infos_dict

The function scan_databases scans for available databases based on specified formats and releases.

Parameters

database_formats: The database_formats parameter is a list that specifies the formats of the databases to be scanned. In this case, the accepted format is "parquet"
database_releases: The database_releases parameter is a list that specifies the releases of the databases to be scanned. In the provided function, the default value for database_releases is set to ["current"], meaning that by default, the function will scan databases that are in the "current"

Returns

The function scan_databases returns a dictionary containing information about databases that match the specified formats and releases.

def annotation(self) -> None: View Source

2730    def annotation(self) -> None:
2731        """
2732        It annotates the VCF file with the annotations specified in the config file.
2733        """
2734
2735        # Config
2736        config = self.get_config()
2737
2738        # Param
2739        param = self.get_param()
2740
2741        # Param - Assembly
2742        assembly = param.get("assembly", config.get("assembly", None))
2743        if not assembly:
2744            assembly = DEFAULT_ASSEMBLY
2745            log.warning(f"Default assembly '{assembly}'")
2746
2747        # annotations databases folders
2748        annotations_databases = set(
2749            config.get("folders", {})
2750            .get("databases", {})
2751            .get("annotations", [DEFAULT_ANNOTATIONS_FOLDER])
2752            + config.get("folders", {})
2753            .get("databases", {})
2754            .get("parquet", ["~/howard/databases/parquet/current"])
2755            + config.get("folders", {})
2756            .get("databases", {})
2757            .get("bcftools", ["~/howard/databases/bcftools/current"])
2758        )
2759
2760        # Get param annotations
2761        if param.get("annotations", None) and isinstance(
2762            param.get("annotations", None), str
2763        ):
2764            log.debug(param.get("annotations", None))
2765            param_annotation_list = param.get("annotations").split(",")
2766        else:
2767            param_annotation_list = []
2768
2769        # Each tools param
2770        if param.get("annotation_parquet", None) != None:
2771            log.debug(
2772                f"""param.get("annotation_parquet", None)={param.get("annotation_parquet", None)}"""
2773            )
2774            if isinstance(param.get("annotation_parquet", None), list):
2775                param_annotation_list.append(",".join(param.get("annotation_parquet")))
2776            else:
2777                param_annotation_list.append(param.get("annotation_parquet"))
2778        if param.get("annotation_snpsift", None) != None:
2779            if isinstance(param.get("annotation_snpsift", None), list):
2780                param_annotation_list.append(
2781                    "snpsift:"
2782                    + "+".join(param.get("annotation_snpsift")).replace(",", "+")
2783                )
2784            else:
2785                param_annotation_list.append(
2786                    "snpsift:" + param.get("annotation_snpsift").replace(",", "+")
2787                )
2788        if param.get("annotation_snpeff", None) != None:
2789            param_annotation_list.append("snpeff:" + param.get("annotation_snpeff"))
2790        if param.get("annotation_bcftools", None) != None:
2791            if isinstance(param.get("annotation_bcftools", None), list):
2792                param_annotation_list.append(
2793                    "bcftools:"
2794                    + "+".join(param.get("annotation_bcftools")).replace(",", "+")
2795                )
2796            else:
2797                param_annotation_list.append(
2798                    "bcftools:" + param.get("annotation_bcftools").replace(",", "+")
2799                )
2800        if param.get("annotation_annovar", None) != None:
2801            param_annotation_list.append("annovar:" + param.get("annotation_annovar"))
2802        if param.get("annotation_exomiser", None) != None:
2803            param_annotation_list.append("exomiser:" + param.get("annotation_exomiser"))
2804        if param.get("annotation_splice", None) != None:
2805            param_annotation_list.append("splice:" + param.get("annotation_splice"))
2806
2807        # Merge param annotations list
2808        param["annotations"] = ",".join(param_annotation_list)
2809
2810        # debug
2811        log.debug(f"param_annotations={param['annotations']}")
2812
2813        if param.get("annotations"):
2814
2815            # Log
2816            # log.info("Annotations - Check annotation parameters")
2817
2818            if not "annotation" in param:
2819                param["annotation"] = {}
2820
2821            # List of annotations parameters
2822            annotations_list_input = {}
2823            if isinstance(param.get("annotations", None), str):
2824                annotation_file_list = [
2825                    value for value in param.get("annotations", "").split(",")
2826                ]
2827                for annotation_file in annotation_file_list:
2828                    annotations_list_input[annotation_file] = {"INFO": None}
2829            else:
2830                annotations_list_input = param.get("annotations", {})
2831
2832            log.info(f"Quick Annotations:")
2833            for annotation_key in list(annotations_list_input.keys()):
2834                log.info(f"   {annotation_key}")
2835
2836            # List of annotations and associated fields
2837            annotations_list = {}
2838
2839            for annotation_file in annotations_list_input:
2840
2841                # Explode annotations if ALL
2842                if (
2843                    annotation_file.upper() == "ALL"
2844                    or annotation_file.upper().startswith("ALL:")
2845                ):
2846
2847                    # check ALL parameters (formats, releases)
2848                    annotation_file_split = annotation_file.split(":")
2849                    database_formats = "parquet"
2850                    database_releases = "current"
2851                    for annotation_file_option in annotation_file_split[1:]:
2852                        database_all_options_split = annotation_file_option.split("=")
2853                        if database_all_options_split[0] == "format":
2854                            database_formats = database_all_options_split[1].split("+")
2855                        if database_all_options_split[0] == "release":
2856                            database_releases = database_all_options_split[1].split("+")
2857
2858                    # Scan for availabled databases
2859                    databases_infos_dict = self.scan_databases(
2860                        database_formats=database_formats,
2861                        database_releases=database_releases,
2862                    )
2863
2864                    # Add found databases in annotation parameters
2865                    for database_infos in databases_infos_dict.keys():
2866                        annotations_list[database_infos] = {"INFO": None}
2867
2868                else:
2869                    annotations_list[annotation_file] = annotations_list_input[
2870                        annotation_file
2871                    ]
2872
2873            # Check each databases
2874            if len(annotations_list):
2875
2876                log.info(
2877                    f"Annotations - Check annotation parameters - Check {len(annotations_list)} databases..."
2878                )
2879
2880                for annotation_file in annotations_list:
2881
2882                    # Init
2883                    annotations = annotations_list.get(annotation_file, None)
2884
2885                    # Annotation snpEff
2886                    if annotation_file.startswith("snpeff"):
2887
2888                        log.debug(f"Quick Annotation snpEff")
2889
2890                        if "snpeff" not in param["annotation"]:
2891                            param["annotation"]["snpeff"] = {}
2892
2893                        if "options" not in param["annotation"]["snpeff"]:
2894                            param["annotation"]["snpeff"]["options"] = ""
2895
2896                        # snpEff options in annotations
2897                        param["annotation"]["snpeff"]["options"] = "".join(
2898                            annotation_file.split(":")[1:]
2899                        )
2900
2901                    # Annotation Annovar
2902                    elif annotation_file.startswith("annovar"):
2903
2904                        log.debug(f"Quick Annotation Annovar")
2905
2906                        if "annovar" not in param["annotation"]:
2907                            param["annotation"]["annovar"] = {}
2908
2909                        if "annotations" not in param["annotation"]["annovar"]:
2910                            param["annotation"]["annovar"]["annotations"] = {}
2911
2912                        # Options
2913                        annotation_file_split = annotation_file.split(":")
2914                        for annotation_file_annotation in annotation_file_split[1:]:
2915                            if annotation_file_annotation:
2916                                param["annotation"]["annovar"]["annotations"][
2917                                    annotation_file_annotation
2918                                ] = annotations
2919
2920                    # Annotation Exomiser
2921                    elif annotation_file.startswith("exomiser"):
2922
2923                        log.debug(f"Quick Annotation Exomiser")
2924
2925                        param["annotation"]["exomiser"] = params_string_to_dict(
2926                            annotation_file
2927                        )
2928
2929                    # Annotation Splice
2930                    elif annotation_file.startswith("splice"):
2931
2932                        log.debug(f"Quick Annotation Splice")
2933
2934                        param["annotation"]["splice"] = params_string_to_dict(
2935                            annotation_file
2936                        )
2937
2938                    # Annotation Parquet or BCFTOOLS
2939                    else:
2940
2941                        # Tools detection
2942                        if annotation_file.startswith("bcftools:"):
2943                            annotation_tool_initial = "bcftools"
2944                            annotation_file = ":".join(annotation_file.split(":")[1:])
2945                        elif annotation_file.startswith("snpsift:"):
2946                            annotation_tool_initial = "snpsift"
2947                            annotation_file = ":".join(annotation_file.split(":")[1:])
2948                        else:
2949                            annotation_tool_initial = None
2950
2951                        # list of files
2952                        annotation_file_list = annotation_file.replace("+", ":").split(
2953                            ":"
2954                        )
2955
2956                        for annotation_file in annotation_file_list:
2957
2958                            if annotation_file:
2959
2960                                # Annotation tool initial
2961                                annotation_tool = annotation_tool_initial
2962
2963                                # Find file
2964                                annotation_file_found = None
2965
2966                                # Expand user
2967                                annotation_file = full_path(annotation_file)
2968
2969                                if os.path.exists(annotation_file):
2970                                    annotation_file_found = annotation_file
2971
2972                                else:
2973                                    # Find within assembly folders
2974                                    for annotations_database in annotations_databases:
2975                                        found_files = find_all(
2976                                            annotation_file,
2977                                            os.path.join(
2978                                                annotations_database, assembly
2979                                            ),
2980                                        )
2981                                        if len(found_files) > 0:
2982                                            annotation_file_found = found_files[0]
2983                                            break
2984                                    if not annotation_file_found and not assembly:
2985                                        # Find within folders
2986                                        for (
2987                                            annotations_database
2988                                        ) in annotations_databases:
2989                                            found_files = find_all(
2990                                                annotation_file, annotations_database
2991                                            )
2992                                            if len(found_files) > 0:
2993                                                annotation_file_found = found_files[0]
2994                                                break
2995                                log.debug(
2996                                    f"for {annotation_file} annotation_file_found={annotation_file_found}"
2997                                )
2998
2999                                # Full path
3000                                annotation_file_found = full_path(annotation_file_found)
3001
3002                                if annotation_file_found:
3003
3004                                    database = Database(database=annotation_file_found)
3005                                    quick_annotation_format = database.get_format()
3006                                    quick_annotation_is_compressed = (
3007                                        database.is_compressed()
3008                                    )
3009                                    quick_annotation_is_indexed = os.path.exists(
3010                                        f"{annotation_file_found}.tbi"
3011                                    )
3012                                    bcftools_preference = False
3013
3014                                    # Check Annotation Tool
3015                                    if not annotation_tool:
3016                                        if (
3017                                            bcftools_preference
3018                                            and quick_annotation_format
3019                                            in ["vcf", "bed"]
3020                                            and quick_annotation_is_compressed
3021                                            and quick_annotation_is_indexed
3022                                        ):
3023                                            annotation_tool = "bcftools"
3024                                        elif quick_annotation_format in [
3025                                            "vcf",
3026                                            "bed",
3027                                            "tsv",
3028                                            "tsv",
3029                                            "csv",
3030                                            "json",
3031                                            "tbl",
3032                                            "parquet",
3033                                            "duckdb",
3034                                        ]:
3035                                            annotation_tool = "parquet"
3036                                        else:
3037                                            log.error(
3038                                                f"Quick Annotation File {annotation_file_found} - Format {quick_annotation_format} not supported yet"
3039                                            )
3040                                            raise ValueError(
3041                                                f"Quick Annotation File {annotation_file_found} - Format {quick_annotation_format} not supported yet"
3042                                            )
3043
3044                                    log.debug(
3045                                        f"Quick Annotation File {annotation_file} - Annotation tool: {annotation_tool}"
3046                                    )
3047
3048                                    # Annotation Tool dispatch
3049                                    if annotation_tool:
3050                                        if annotation_tool not in param["annotation"]:
3051                                            param["annotation"][annotation_tool] = {}
3052                                        if (
3053                                            "annotations"
3054                                            not in param["annotation"][annotation_tool]
3055                                        ):
3056                                            param["annotation"][annotation_tool][
3057                                                "annotations"
3058                                            ] = {}
3059                                        param["annotation"][annotation_tool][
3060                                            "annotations"
3061                                        ][annotation_file_found] = annotations
3062
3063                                else:
3064                                    log.error(
3065                                        f"Quick Annotation File {annotation_file} does NOT exist"
3066                                    )
3067
3068                self.set_param(param)
3069
3070        if param.get("annotation", None):
3071            log.info("Annotations")
3072            if param.get("annotation", {}).get("parquet", None):
3073                log.info("Annotations 'parquet'...")
3074                self.annotation_parquet()
3075            if param.get("annotation", {}).get("bcftools", None):
3076                log.info("Annotations 'bcftools'...")
3077                self.annotation_bcftools()
3078            if param.get("annotation", {}).get("snpsift", None):
3079                log.info("Annotations 'snpsift'...")
3080                self.annotation_snpsift()
3081            if param.get("annotation", {}).get("annovar", None):
3082                log.info("Annotations 'annovar'...")
3083                self.annotation_annovar()
3084            if param.get("annotation", {}).get("snpeff", None):
3085                log.info("Annotations 'snpeff'...")
3086                self.annotation_snpeff()
3087            if param.get("annotation", {}).get("exomiser", None) is not None:
3088                log.info("Annotations 'exomiser'...")
3089                self.annotation_exomiser()
3090            if param.get("annotation", {}).get("splice", None) is not None:
3091                log.info("Annotations 'splice' ...")
3092                self.annotation_splice()
3093
3094        # Explode INFOS fields into table fields
3095        if self.get_explode_infos():
3096            self.explode_infos(
3097                prefix=self.get_explode_infos_prefix(),
3098                fields=self.get_explode_infos_fields(),
3099                force=True,
3100            )

It annotates the VCF file with the annotations specified in the config file.

def annotation_snpsift(self, threads: int = None) -> None: View Source

3102    def annotation_snpsift(self, threads: int = None) -> None:
3103        """
3104        This function annotate with bcftools
3105
3106        :param threads: Number of threads to use
3107        :return: the value of the variable "return_value".
3108        """
3109
3110        # DEBUG
3111        log.debug("Start annotation with bcftools databases")
3112
3113        # Threads
3114        if not threads:
3115            threads = self.get_threads()
3116        log.debug("Threads: " + str(threads))
3117
3118        # Config
3119        config = self.get_config()
3120        log.debug("Config: " + str(config))
3121
3122        # Config - snpSift
3123        snpsift_bin_command = get_bin_command(
3124            bin="SnpSift.jar",
3125            tool="snpsift",
3126            bin_type="jar",
3127            config=config,
3128            default_folder=f"{DEFAULT_TOOLS_FOLDER}/snpeff",
3129        )
3130        if not snpsift_bin_command:
3131            msg_err = f"Annotation failed: no snpsift bin '{snpsift_bin_command}'"
3132            log.error(msg_err)
3133            raise ValueError(msg_err)
3134
3135        # Config - bcftools
3136        bcftools_bin_command = get_bin_command(
3137            bin="bcftools",
3138            tool="bcftools",
3139            bin_type="bin",
3140            config=config,
3141            default_folder=f"{DEFAULT_TOOLS_FOLDER}/bcftools",
3142        )
3143        if not bcftools_bin_command:
3144            msg_err = f"Annotation failed: no bcftools bin '{bcftools_bin_command}'"
3145            log.error(msg_err)
3146            raise ValueError(msg_err)
3147
3148        # Config - BCFTools databases folders
3149        databases_folders = set(
3150            self.get_config()
3151            .get("folders", {})
3152            .get("databases", {})
3153            .get("annotations", ["."])
3154            + self.get_config()
3155            .get("folders", {})
3156            .get("databases", {})
3157            .get("bcftools", ["."])
3158        )
3159        log.debug("Databases annotations: " + str(databases_folders))
3160
3161        # Param
3162        annotations = (
3163            self.get_param()
3164            .get("annotation", {})
3165            .get("snpsift", {})
3166            .get("annotations", None)
3167        )
3168        log.debug("Annotations: " + str(annotations))
3169
3170        # Assembly
3171        assembly = self.get_param().get(
3172            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
3173        )
3174
3175        # Data
3176        table_variants = self.get_table_variants()
3177
3178        # Check if not empty
3179        log.debug("Check if not empty")
3180        sql_query_chromosomes = (
3181            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
3182        )
3183        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
3184        if not sql_query_chromosomes_df["count"][0]:
3185            log.info(f"VCF empty")
3186            return
3187
3188        # VCF header
3189        vcf_reader = self.get_header()
3190        log.debug("Initial header: " + str(vcf_reader.infos))
3191
3192        # Existing annotations
3193        for vcf_annotation in self.get_header().infos:
3194
3195            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
3196            log.debug(
3197                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
3198            )
3199
3200        if annotations:
3201
3202            with TemporaryDirectory(dir=self.get_tmp_dir()) as tmp_dir:
3203
3204                # Export VCF file
3205                tmp_vcf_name = os.path.join(tmp_dir, "input.vcf.gz")
3206
3207                # Init
3208                commands = {}
3209
3210                for annotation in annotations:
3211                    annotation_fields = annotations[annotation]
3212
3213                    # Annotation Name
3214                    annotation_name = os.path.basename(annotation)
3215
3216                    if not annotation_fields:
3217                        annotation_fields = {"INFO": None}
3218
3219                    log.debug(f"Annotation '{annotation_name}'")
3220                    log.debug(
3221                        f"Annotation '{annotation_name}' - fields: {annotation_fields}"
3222                    )
3223
3224                    # Create Database
3225                    database = Database(
3226                        database=annotation,
3227                        databases_folders=databases_folders,
3228                        assembly=assembly,
3229                    )
3230
3231                    # Find files
3232                    db_file = database.get_database()
3233                    db_file = full_path(db_file)
3234                    db_hdr_file = database.get_header_file()
3235                    db_hdr_file = full_path(db_hdr_file)
3236                    db_file_type = database.get_format()
3237                    db_tbi_file = f"{db_file}.tbi"
3238                    db_file_compressed = database.is_compressed()
3239
3240                    # Check if compressed
3241                    if not db_file_compressed:
3242                        log.error(
3243                            f"Annotation '{annotation}' - {db_file} NOT compressed file"
3244                        )
3245                        raise ValueError(
3246                            f"Annotation '{annotation}' - {db_file} NOT compressed file"
3247                        )
3248
3249                    # Check if indexed
3250                    if not os.path.exists(db_tbi_file):
3251                        log.error(
3252                            f"Annotation '{annotation}' - {db_file} NOT indexed file"
3253                        )
3254                        raise ValueError(
3255                            f"Annotation '{annotation}' - {db_file} NOT indexed file"
3256                        )
3257
3258                    # Check index - try to create if not exists
3259                    if not os.path.exists(db_file) or not os.path.exists(db_hdr_file):
3260                        log.error("Annotation failed: database not valid")
3261                        log.error(f"Annotation annotation file: {db_file}")
3262                        log.error(f"Annotation annotation header: {db_hdr_file}")
3263                        log.error(f"Annotation annotation index: {db_tbi_file}")
3264                        raise ValueError(
3265                            f"Annotation failed: database not valid - annotation file {db_file} / annotation header {db_hdr_file} / annotation index {db_tbi_file} / annotation compression {db_file_compressed}"
3266                        )
3267                    else:
3268
3269                        log.debug(
3270                            f"Annotation '{annotation}' - file: "
3271                            + str(db_file)
3272                            + " and "
3273                            + str(db_hdr_file)
3274                        )
3275
3276                        # Load header as VCF object
3277                        db_hdr_vcf = Variants(input=db_hdr_file)
3278                        db_hdr_vcf_header_infos = db_hdr_vcf.get_header().infos
3279                        log.debug(
3280                            "Annotation database header: "
3281                            + str(db_hdr_vcf_header_infos)
3282                        )
3283
3284                        # For all fields in database
3285                        annotation_fields_full = False
3286                        if "ALL" in annotation_fields or "INFO" in annotation_fields:
3287                            annotation_fields = {
3288                                key: key for key in db_hdr_vcf_header_infos
3289                            }
3290                            log.debug(
3291                                "Annotation database header - All annotations added: "
3292                                + str(annotation_fields)
3293                            )
3294                            annotation_fields_full = True
3295
3296                        # # Create file for field rename
3297                        # log.debug("Create file for field rename")
3298                        # tmp_rename = NamedTemporaryFile(
3299                        #     prefix=self.get_prefix(),
3300                        #     dir=self.get_tmp_dir(),
3301                        #     suffix=".rename",
3302                        #     delete=False,
3303                        # )
3304                        # tmp_rename_name = tmp_rename.name
3305                        # tmp_files.append(tmp_rename_name)
3306
3307                        # Number of fields
3308                        nb_annotation_field = 0
3309                        annotation_list = []
3310                        annotation_infos_rename_list = []
3311
3312                        for annotation_field in annotation_fields:
3313
3314                            # field new name, if parametered SKIPPED !!!!!! not managed actually TODO
3315                            annotation_fields_new_name = annotation_fields.get(
3316                                annotation_field, annotation_field
3317                            )
3318                            if not annotation_fields_new_name:
3319                                annotation_fields_new_name = annotation_field
3320
3321                            # Check if field is in DB and if field is not elready in input data
3322                            if (
3323                                annotation_field in db_hdr_vcf.get_header().infos
3324                                and annotation_fields_new_name
3325                                not in self.get_header().infos
3326                            ):
3327
3328                                log.info(
3329                                    f"Annotation '{annotation_name}' - '{annotation_field}' -> '{annotation_fields_new_name}'"
3330                                )
3331
3332                                # BCFTools annotate param to rename fields
3333                                if annotation_field != annotation_fields_new_name:
3334                                    annotation_infos_rename_list.append(
3335                                        f"{annotation_fields_new_name}:=INFO/{annotation_field}"
3336                                    )
3337
3338                                # Add INFO field to header
3339                                db_hdr_vcf_header_infos_number = (
3340                                    db_hdr_vcf_header_infos[annotation_field].num or "."
3341                                )
3342                                db_hdr_vcf_header_infos_type = (
3343                                    db_hdr_vcf_header_infos[annotation_field].type
3344                                    or "String"
3345                                )
3346                                db_hdr_vcf_header_infos_description = (
3347                                    db_hdr_vcf_header_infos[annotation_field].desc
3348                                    or f"{annotation_field} description"
3349                                )
3350                                db_hdr_vcf_header_infos_source = (
3351                                    db_hdr_vcf_header_infos[annotation_field].source
3352                                    or "unknown"
3353                                )
3354                                db_hdr_vcf_header_infos_version = (
3355                                    db_hdr_vcf_header_infos[annotation_field].version
3356                                    or "unknown"
3357                                )
3358
3359                                vcf_reader.infos[annotation_fields_new_name] = (
3360                                    vcf.parser._Info(
3361                                        annotation_fields_new_name,
3362                                        db_hdr_vcf_header_infos_number,
3363                                        db_hdr_vcf_header_infos_type,
3364                                        db_hdr_vcf_header_infos_description,
3365                                        db_hdr_vcf_header_infos_source,
3366                                        db_hdr_vcf_header_infos_version,
3367                                        self.code_type_map[
3368                                            db_hdr_vcf_header_infos_type
3369                                        ],
3370                                    )
3371                                )
3372
3373                                annotation_list.append(annotation_field)
3374
3375                                nb_annotation_field += 1
3376
3377                            else:
3378
3379                                if (
3380                                    annotation_field
3381                                    not in db_hdr_vcf.get_header().infos
3382                                ):
3383                                    log.warning(
3384                                        f"Annotation '{annotation_name}' - '{annotation_field}' - not available in vcf/bed file"
3385                                    )
3386                                if (
3387                                    annotation_fields_new_name
3388                                    in self.get_header().infos
3389                                ):
3390                                    log.warning(
3391                                        f"Annotation '{annotation_name}' - '{annotation_fields_new_name}' - already exists (skipped)"
3392                                    )
3393
3394                        log.info(
3395                            f"Annotation '{annotation_name}' - {nb_annotation_field} annotations available in vcf/bed file"
3396                        )
3397
3398                        annotation_infos = ",".join(annotation_list)
3399
3400                        if annotation_infos != "":
3401
3402                            # Annotated VCF (and error file)
3403                            tmp_annotation_vcf_name = os.path.join(
3404                                tmp_dir, os.path.basename(annotation) + ".vcf.gz"
3405                            )
3406                            tmp_annotation_vcf_name_err = (
3407                                tmp_annotation_vcf_name + ".err"
3408                            )
3409
3410                            # Add fields to annotate
3411                            if not annotation_fields_full:
3412                                annotation_infos_option = f"-info {annotation_infos}"
3413                            else:
3414                                annotation_infos_option = ""
3415
3416                            # Info fields rename
3417                            if annotation_infos_rename_list:
3418                                annotation_infos_rename = " -c " + ",".join(
3419                                    annotation_infos_rename_list
3420                                )
3421                            else:
3422                                annotation_infos_rename = ""
3423
3424                            # Annotate command
3425                            command_annotate = f"{snpsift_bin_command} annotate {annotation_infos_option} {db_file} {tmp_vcf_name} | {bcftools_bin_command} annotate --threads={threads} {annotation_infos_rename} -Oz1 -o {tmp_annotation_vcf_name} 2>>{tmp_annotation_vcf_name_err} "
3426
3427                            # Add command
3428                            commands[command_annotate] = tmp_annotation_vcf_name
3429
3430                if commands:
3431
3432                    # Export VCF file
3433                    self.export_variant_vcf(
3434                        vcf_file=tmp_vcf_name,
3435                        remove_info=True,
3436                        add_samples=False,
3437                        index=True,
3438                    )
3439                    shutil.copyfile(tmp_vcf_name, "/tmp/input.vcf")
3440
3441                    # Num command
3442                    nb_command = 0
3443
3444                    # Annotate
3445                    for command_annotate in commands:
3446                        nb_command += 1
3447                        log.info(
3448                            f"Annotation - Annotate [{nb_command}/{len(commands)}]..."
3449                        )
3450                        log.debug(f"command_annotate={command_annotate}")
3451                        run_parallel_commands([command_annotate], threads)
3452
3453                        # Debug
3454                        shutil.copyfile(commands[command_annotate], "/tmp/snpsift.vcf")
3455
3456                        # Update variants
3457                        log.info(
3458                            f"Annotation - Updating [{nb_command}/{len(commands)}]..."
3459                        )
3460                        self.update_from_vcf(commands[command_annotate])

This function annotate with bcftools

Parameters

threads: Number of threads to use

Returns

the value of the variable "return_value".

def annotation_bcftools(self, threads: int = None) -> None: View Source

3462    def annotation_bcftools(self, threads: int = None) -> None:
3463        """
3464        This function annotate with bcftools
3465
3466        :param threads: Number of threads to use
3467        :return: the value of the variable "return_value".
3468        """
3469
3470        # DEBUG
3471        log.debug("Start annotation with bcftools databases")
3472
3473        # Threads
3474        if not threads:
3475            threads = self.get_threads()
3476        log.debug("Threads: " + str(threads))
3477
3478        # Config
3479        config = self.get_config()
3480        log.debug("Config: " + str(config))
3481
3482        # DEBUG
3483        delete_tmp = True
3484        if self.get_config().get("verbosity", "warning") in ["debug"]:
3485            delete_tmp = False
3486            log.debug("Delete tmp files/folders: " + str(delete_tmp))
3487
3488        # Config - BCFTools bin command
3489        bcftools_bin_command = get_bin_command(
3490            bin="bcftools",
3491            tool="bcftools",
3492            bin_type="bin",
3493            config=config,
3494            default_folder=f"{DEFAULT_TOOLS_FOLDER}/bcftools",
3495        )
3496        if not bcftools_bin_command:
3497            msg_err = f"Annotation failed: no bcftools bin '{bcftools_bin_command}'"
3498            log.error(msg_err)
3499            raise ValueError(msg_err)
3500
3501        # Config - BCFTools databases folders
3502        databases_folders = set(
3503            self.get_config()
3504            .get("folders", {})
3505            .get("databases", {})
3506            .get("annotations", ["."])
3507            + self.get_config()
3508            .get("folders", {})
3509            .get("databases", {})
3510            .get("bcftools", ["."])
3511        )
3512        log.debug("Databases annotations: " + str(databases_folders))
3513
3514        # Param
3515        annotations = (
3516            self.get_param()
3517            .get("annotation", {})
3518            .get("bcftools", {})
3519            .get("annotations", None)
3520        )
3521        log.debug("Annotations: " + str(annotations))
3522
3523        # Assembly
3524        assembly = self.get_param().get(
3525            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
3526        )
3527
3528        # Data
3529        table_variants = self.get_table_variants()
3530
3531        # Check if not empty
3532        log.debug("Check if not empty")
3533        sql_query_chromosomes = (
3534            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
3535        )
3536        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
3537        if not sql_query_chromosomes_df["count"][0]:
3538            log.info(f"VCF empty")
3539            return
3540
3541        # Export in VCF
3542        log.debug("Create initial file to annotate")
3543        tmp_vcf = NamedTemporaryFile(
3544            prefix=self.get_prefix(),
3545            dir=self.get_tmp_dir(),
3546            suffix=".vcf.gz",
3547            delete=False,
3548        )
3549        tmp_vcf_name = tmp_vcf.name
3550
3551        # VCF header
3552        vcf_reader = self.get_header()
3553        log.debug("Initial header: " + str(vcf_reader.infos))
3554
3555        # Existing annotations
3556        for vcf_annotation in self.get_header().infos:
3557
3558            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
3559            log.debug(
3560                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
3561            )
3562
3563        if annotations:
3564
3565            tmp_ann_vcf_list = []
3566            commands = []
3567            tmp_files = []
3568            err_files = []
3569
3570            for annotation in annotations:
3571                annotation_fields = annotations[annotation]
3572
3573                # Annotation Name
3574                annotation_name = os.path.basename(annotation)
3575
3576                if not annotation_fields:
3577                    annotation_fields = {"INFO": None}
3578
3579                log.debug(f"Annotation '{annotation_name}'")
3580                log.debug(
3581                    f"Annotation '{annotation_name}' - fields: {annotation_fields}"
3582                )
3583
3584                # Create Database
3585                database = Database(
3586                    database=annotation,
3587                    databases_folders=databases_folders,
3588                    assembly=assembly,
3589                )
3590
3591                # Find files
3592                db_file = database.get_database()
3593                db_file = full_path(db_file)
3594                db_hdr_file = database.get_header_file()
3595                db_hdr_file = full_path(db_hdr_file)
3596                db_file_type = database.get_format()
3597                db_tbi_file = f"{db_file}.tbi"
3598                db_file_compressed = database.is_compressed()
3599
3600                # Check if compressed
3601                if not db_file_compressed:
3602                    log.error(
3603                        f"Annotation '{annotation}' - {db_file} NOT compressed file"
3604                    )
3605                    raise ValueError(
3606                        f"Annotation '{annotation}' - {db_file} NOT compressed file"
3607                    )
3608
3609                # Check if indexed
3610                if not os.path.exists(db_tbi_file):
3611                    log.error(f"Annotation '{annotation}' - {db_file} NOT indexed file")
3612                    raise ValueError(
3613                        f"Annotation '{annotation}' - {db_file} NOT indexed file"
3614                    )
3615
3616                # Check index - try to create if not exists
3617                if not os.path.exists(db_file) or not os.path.exists(db_hdr_file):
3618                    log.error("Annotation failed: database not valid")
3619                    log.error(f"Annotation annotation file: {db_file}")
3620                    log.error(f"Annotation annotation header: {db_hdr_file}")
3621                    log.error(f"Annotation annotation index: {db_tbi_file}")
3622                    raise ValueError(
3623                        f"Annotation failed: database not valid - annotation file {db_file} / annotation header {db_hdr_file} / annotation index {db_tbi_file} / annotation compression {db_file_compressed}"
3624                    )
3625                else:
3626
3627                    log.debug(
3628                        f"Annotation '{annotation}' - file: "
3629                        + str(db_file)
3630                        + " and "
3631                        + str(db_hdr_file)
3632                    )
3633
3634                    # Load header as VCF object
3635                    db_hdr_vcf = Variants(input=db_hdr_file)
3636                    db_hdr_vcf_header_infos = db_hdr_vcf.get_header().infos
3637                    log.debug(
3638                        "Annotation database header: " + str(db_hdr_vcf_header_infos)
3639                    )
3640
3641                    # For all fields in database
3642                    if "ALL" in annotation_fields or "INFO" in annotation_fields:
3643                        annotation_fields = {
3644                            key: key for key in db_hdr_vcf_header_infos
3645                        }
3646                        log.debug(
3647                            "Annotation database header - All annotations added: "
3648                            + str(annotation_fields)
3649                        )
3650
3651                    # Number of fields
3652                    nb_annotation_field = 0
3653                    annotation_list = []
3654
3655                    for annotation_field in annotation_fields:
3656
3657                        # field new name, if parametered SKIPPED !!!!!! not managed actually TODO
3658                        annotation_fields_new_name = annotation_fields.get(
3659                            annotation_field, annotation_field
3660                        )
3661                        if not annotation_fields_new_name:
3662                            annotation_fields_new_name = annotation_field
3663
3664                        # Check if field is in DB and if field is not elready in input data
3665                        if (
3666                            annotation_field in db_hdr_vcf.get_header().infos
3667                            and annotation_fields_new_name
3668                            not in self.get_header().infos
3669                        ):
3670
3671                            log.info(
3672                                f"Annotation '{annotation_name}' - '{annotation_field}' -> '{annotation_fields_new_name}'"
3673                            )
3674
3675                            # Add INFO field to header
3676                            db_hdr_vcf_header_infos_number = (
3677                                db_hdr_vcf_header_infos[annotation_field].num or "."
3678                            )
3679                            db_hdr_vcf_header_infos_type = (
3680                                db_hdr_vcf_header_infos[annotation_field].type
3681                                or "String"
3682                            )
3683                            db_hdr_vcf_header_infos_description = (
3684                                db_hdr_vcf_header_infos[annotation_field].desc
3685                                or f"{annotation_field} description"
3686                            )
3687                            db_hdr_vcf_header_infos_source = (
3688                                db_hdr_vcf_header_infos[annotation_field].source
3689                                or "unknown"
3690                            )
3691                            db_hdr_vcf_header_infos_version = (
3692                                db_hdr_vcf_header_infos[annotation_field].version
3693                                or "unknown"
3694                            )
3695
3696                            vcf_reader.infos[annotation_fields_new_name] = (
3697                                vcf.parser._Info(
3698                                    annotation_fields_new_name,
3699                                    db_hdr_vcf_header_infos_number,
3700                                    db_hdr_vcf_header_infos_type,
3701                                    db_hdr_vcf_header_infos_description,
3702                                    db_hdr_vcf_header_infos_source,
3703                                    db_hdr_vcf_header_infos_version,
3704                                    self.code_type_map[db_hdr_vcf_header_infos_type],
3705                                )
3706                            )
3707
3708                            # annotation_list.append(annotation_field)
3709                            if annotation_field != annotation_fields_new_name:
3710                                annotation_list.append(
3711                                    f"{annotation_fields_new_name}:=INFO/{annotation_field}"
3712                                )
3713                            else:
3714                                annotation_list.append(annotation_field)
3715
3716                            nb_annotation_field += 1
3717
3718                        else:
3719
3720                            if annotation_field not in db_hdr_vcf.get_header().infos:
3721                                log.warning(
3722                                    f"Annotation '{annotation}' - '{annotation_field}' - not available in vcf/bed file"
3723                                )
3724                            if annotation_fields_new_name in self.get_header().infos:
3725                                log.warning(
3726                                    f"Annotation '{annotation}' - '{annotation_fields_new_name}' - already exists (skipped)"
3727                                )
3728
3729                    log.info(
3730                        f"Annotation '{annotation_name}' - {nb_annotation_field} annotations available in vcf/bed file"
3731                    )
3732
3733                    annotation_infos = ",".join(annotation_list)
3734
3735                    if annotation_infos != "":
3736
3737                        # Protect header for bcftools (remove "#CHROM" and variants line)
3738                        log.debug("Protect Header file - remove #CHROM line if exists")
3739                        tmp_header_vcf = NamedTemporaryFile(
3740                            prefix=self.get_prefix(),
3741                            dir=self.get_tmp_dir(),
3742                            suffix=".hdr",
3743                            delete=False,
3744                        )
3745                        tmp_header_vcf_name = tmp_header_vcf.name
3746                        tmp_files.append(tmp_header_vcf_name)
3747                        # Command
3748                        if db_hdr_file.endswith(".gz"):
3749                            command_extract_header = f"zcat {db_hdr_file} | grep '^##' > {tmp_header_vcf_name}"
3750                        else:
3751                            command_extract_header = f"cat {db_hdr_file} | grep '^##' > {tmp_header_vcf_name}"
3752                        # Run
3753                        run_parallel_commands([command_extract_header], 1)
3754
3755                        # Find chomosomes
3756                        log.debug("Find chromosomes ")
3757                        sql_query_chromosomes = f"""SELECT table_variants.\"#CHROM\" as CHROM FROM {table_variants} as table_variants GROUP BY table_variants.\"#CHROM\""""
3758                        sql_query_chromosomes_df = self.get_query_to_df(
3759                            sql_query_chromosomes
3760                        )
3761                        chomosomes_list = list(sql_query_chromosomes_df["CHROM"])
3762
3763                        log.debug("Chromosomes found: " + str(list(chomosomes_list)))
3764
3765                        # BED columns in the annotation file
3766                        if db_file_type in ["bed"]:
3767                            annotation_infos = "CHROM,POS,POS," + annotation_infos
3768
3769                        for chrom in chomosomes_list:
3770
3771                            # Create BED on initial VCF
3772                            log.debug("Create BED on initial VCF: " + str(tmp_vcf_name))
3773                            tmp_bed = NamedTemporaryFile(
3774                                prefix=self.get_prefix(),
3775                                dir=self.get_tmp_dir(),
3776                                suffix=".bed",
3777                                delete=False,
3778                            )
3779                            tmp_bed_name = tmp_bed.name
3780                            tmp_files.append(tmp_bed_name)
3781
3782                            # Detecte regions
3783                            log.debug(
3784                                f"Annotation '{annotation}' - Chromosome '{chrom}' - Start detecting regions..."
3785                            )
3786                            window = 1000000
3787                            sql_query_intervals_for_bed = f"""
3788                                SELECT  \"#CHROM\",
3789                                        CASE WHEN \"POS\"-{window}-1 < 0 THEN 0 ELSE \"POS\"-{window}-1 END,
3790                                        \"POS\"+{window}
3791                                FROM {table_variants} as table_variants
3792                                WHERE table_variants.\"#CHROM\" = '{chrom}'
3793                            """
3794                            regions = self.conn.execute(
3795                                sql_query_intervals_for_bed
3796                            ).fetchall()
3797                            merged_regions = merge_regions(regions)
3798                            log.debug(
3799                                f"Annotation '{annotation}' - Chromosome '{chrom}' - Stop detecting regions..."
3800                            )
3801
3802                            header = ["#CHROM", "START", "END"]
3803                            with open(tmp_bed_name, "w") as f:
3804                                # Write the header with tab delimiter
3805                                f.write("\t".join(header) + "\n")
3806                                for d in merged_regions:
3807                                    # Write each data row with tab delimiter
3808                                    f.write("\t".join(map(str, d)) + "\n")
3809
3810                            # Tmp files
3811                            tmp_annotation_vcf = NamedTemporaryFile(
3812                                prefix=self.get_prefix(),
3813                                dir=self.get_tmp_dir(),
3814                                suffix=".vcf.gz",
3815                                delete=False,
3816                            )
3817                            tmp_annotation_vcf_name = tmp_annotation_vcf.name
3818                            tmp_files.append(tmp_annotation_vcf_name)
3819                            tmp_ann_vcf_list.append(f"{tmp_annotation_vcf_name}")
3820                            tmp_annotation_vcf_name_err = (
3821                                tmp_annotation_vcf_name + ".err"
3822                            )
3823                            err_files.append(tmp_annotation_vcf_name_err)
3824
3825                            # Annotate Command
3826                            log.debug(
3827                                f"Annotation '{annotation}' - add bcftools command"
3828                            )
3829
3830                            # Command
3831                            command_annotate = f"{bcftools_bin_command} annotate --pair-logic exact --regions-file={tmp_bed_name} -a {db_file} -h {tmp_header_vcf_name} -c {annotation_infos} {tmp_vcf_name} -o {tmp_annotation_vcf_name} -Oz1 2>>{tmp_annotation_vcf_name_err} && tabix {tmp_annotation_vcf_name} 2>>{tmp_annotation_vcf_name_err} "
3832
3833                            # Add command
3834                            commands.append(command_annotate)
3835
3836            # if some commands
3837            if commands:
3838
3839                # Export VCF file
3840                self.export_variant_vcf(
3841                    vcf_file=tmp_vcf_name,
3842                    remove_info=True,
3843                    add_samples=False,
3844                    index=True,
3845                )
3846
3847                # Threads
3848                # calculate threads for annotated commands
3849                if commands:
3850                    threads_bcftools_annotate = round(threads / len(commands))
3851                else:
3852                    threads_bcftools_annotate = 1
3853
3854                if not threads_bcftools_annotate:
3855                    threads_bcftools_annotate = 1
3856
3857                # Add threads option to bcftools commands
3858                if threads_bcftools_annotate > 1:
3859                    commands_threaded = []
3860                    for command in commands:
3861                        commands_threaded.append(
3862                            command.replace(
3863                                f"{bcftools_bin_command} annotate ",
3864                                f"{bcftools_bin_command} annotate --threads={threads_bcftools_annotate} ",
3865                            )
3866                        )
3867                    commands = commands_threaded
3868
3869                # Command annotation multithreading
3870                log.debug(f"Annotation - Annotation commands: " + str(commands))
3871                log.info(
3872                    f"Annotation - Annotation multithreaded in "
3873                    + str(len(commands))
3874                    + " commands"
3875                )
3876
3877                run_parallel_commands(commands, threads)
3878
3879                # Merge
3880                tmp_ann_vcf_list_cmd = " ".join(tmp_ann_vcf_list)
3881
3882                if tmp_ann_vcf_list_cmd:
3883
3884                    # Tmp file
3885                    tmp_annotate_vcf = NamedTemporaryFile(
3886                        prefix=self.get_prefix(),
3887                        dir=self.get_tmp_dir(),
3888                        suffix=".vcf.gz",
3889                        delete=True,
3890                    )
3891                    tmp_annotate_vcf_name = tmp_annotate_vcf.name
3892                    tmp_annotate_vcf_name_err = tmp_annotate_vcf_name + ".err"
3893                    err_files.append(tmp_annotate_vcf_name_err)
3894
3895                    # Tmp file remove command
3896                    tmp_files_remove_command = ""
3897                    if tmp_files:
3898                        tmp_files_remove_command = " && rm -f " + " ".join(tmp_files)
3899
3900                    # Command merge
3901                    merge_command = f"{bcftools_bin_command} merge --force-samples --threads={threads} {tmp_vcf_name} {tmp_ann_vcf_list_cmd} -o {tmp_annotate_vcf_name} -Oz 2>>{tmp_annotate_vcf_name_err} {tmp_files_remove_command}"
3902                    log.info(
3903                        f"Annotation - Annotation merging "
3904                        + str(len(commands))
3905                        + " annotated files"
3906                    )
3907                    log.debug(f"Annotation - merge command: {merge_command}")
3908                    run_parallel_commands([merge_command], 1)
3909
3910                    # Error messages
3911                    log.info(f"Error/Warning messages:")
3912                    error_message_command_all = []
3913                    error_message_command_warning = []
3914                    error_message_command_err = []
3915                    for err_file in err_files:
3916                        with open(err_file, "r") as f:
3917                            for line in f:
3918                                message = line.strip()
3919                                error_message_command_all.append(message)
3920                                if line.startswith("[W::"):
3921                                    error_message_command_warning.append(message)
3922                                if line.startswith("[E::"):
3923                                    error_message_command_err.append(
3924                                        f"{err_file}: " + message
3925                                    )
3926                    # log info
3927                    for message in list(
3928                        set(error_message_command_err + error_message_command_warning)
3929                    ):
3930                        log.info(f"   {message}")
3931                    # debug info
3932                    for message in list(set(error_message_command_all)):
3933                        log.debug(f"   {message}")
3934                    # failed
3935                    if len(error_message_command_err):
3936                        log.error("Annotation failed: Error in commands")
3937                        raise ValueError("Annotation failed: Error in commands")
3938
3939                    # Update variants
3940                    log.info(f"Annotation - Updating...")
3941                    self.update_from_vcf(tmp_annotate_vcf_name)

This function annotate with bcftools

Parameters

threads: Number of threads to use

Returns

the value of the variable "return_value".

def annotation_exomiser(self, threads: int = None) -> None: View Source

3943    def annotation_exomiser(self, threads: int = None) -> None:
3944        """
3945        This function annotate with Exomiser
3946
3947        This function uses args as parameters, in section "annotation" -> "exomiser", with sections:
3948        - "analysis" (dict/file):
3949            Full analysis dictionnary parameters (see Exomiser docs).
3950            Either a dict, or a file in JSON or YAML format.
3951            These parameters may change depending on other parameters (e.g. phenotipicFeatures/HPO)
3952            Default : None
3953        - "preset" (string):
3954            Analysis preset (available in config folder).
3955            Used if no full "analysis" is provided.
3956            Default: "exome"
3957        - "phenopacket" (dict/file):
3958            Samples and phenotipic features parameters (see Exomiser docs).
3959            Either a dict, or a file in JSON or YAML format.
3960            Default: None
3961        - "subject" (dict):
3962            Sample parameters (see Exomiser docs).
3963            Example:
3964                "subject":
3965                    {
3966                        "id": "ISDBM322017",
3967                        "sex": "FEMALE"
3968                    }
3969            Default: None
3970        - "sample" (string):
3971            Sample name to construct "subject" section:
3972                "subject":
3973                    {
3974                        "id": "<sample>",
3975                        "sex": "UNKNOWN_SEX"
3976                    }
3977            Default: None
3978        - "phenotypicFeatures" (dict)
3979            Phenotypic features to construct "subject" section.
3980            Example:
3981                "phenotypicFeatures":
3982                    [
3983                        { "type": { "id": "HP:0001159", "label": "Syndactyly" } },
3984                        { "type": { "id": "HP:0000486", "label": "Strabismus" } }
3985                    ]
3986        - "hpo" (list)
3987            List of HPO ids as phenotypic features.
3988            Example:
3989                "hpo": ['0001156', '0001363', '0011304', '0010055']
3990            Default: []
3991        - "outputOptions" (dict):
3992            Output options (see Exomiser docs).
3993            Default:
3994                "output_options" =
3995                    {
3996                        "outputContributingVariantsOnly": False,
3997                        "numGenes": 0,
3998                        "outputFormats": ["TSV_VARIANT", "VCF"]
3999                    }
4000        - "transcript_source" (string):
4001            Transcript source (either "refseq", "ucsc", "ensembl")
4002            Default: "refseq"
4003        - "exomiser_to_info" (boolean):
4004            Add exomiser TSV file columns as INFO fields in VCF.
4005            Default: False
4006        - "release" (string):
4007            Exomise database release.
4008            If not exists, database release will be downloaded (take a while).
4009            Default: None (provided by application.properties configuration file)
4010        - "exomiser_application_properties" (file):
4011            Exomiser configuration file (see Exomiser docs).
4012            Useful to automatically download databases (especially for specific genome databases).
4013
4014        Notes:
4015        - If no sample in parameters, first sample in VCF will be chosen
4016        - If no HPO found, "hiPhivePrioritiser" analysis step will be switch off
4017
4018        :param threads: The number of threads to use
4019        :return: None.
4020        """
4021
4022        # DEBUG
4023        log.debug("Start annotation with Exomiser databases")
4024
4025        # Threads
4026        if not threads:
4027            threads = self.get_threads()
4028        log.debug("Threads: " + str(threads))
4029
4030        # Config
4031        config = self.get_config()
4032        log.debug("Config: " + str(config))
4033
4034        # Config - Folders - Databases
4035        databases_folders = (
4036            config.get("folders", {})
4037            .get("databases", {})
4038            .get("exomiser", f"{DEFAULT_DATABASE_FOLDER}/exomiser/current")
4039        )
4040        databases_folders = full_path(databases_folders)
4041        if not os.path.exists(databases_folders):
4042            log.error(f"Databases annotations: {databases_folders} NOT found")
4043        log.debug("Databases annotations: " + str(databases_folders))
4044
4045        # Config - Exomiser
4046        exomiser_bin_command = get_bin_command(
4047            bin="exomiser-cli*.jar",
4048            tool="exomiser",
4049            bin_type="jar",
4050            config=config,
4051            default_folder=f"{DEFAULT_TOOLS_FOLDER}/exomiser",
4052        )
4053        log.debug("Exomiser bin command: " + str(exomiser_bin_command))
4054        if not exomiser_bin_command:
4055            msg_err = f"Annotation failed: no exomiser bin '{exomiser_bin_command}'"
4056            log.error(msg_err)
4057            raise ValueError(msg_err)
4058
4059        # Param
4060        param = self.get_param()
4061        log.debug("Param: " + str(param))
4062
4063        # Param - Exomiser
4064        param_exomiser = param.get("annotation", {}).get("exomiser", {})
4065        log.debug(f"Param Exomiser: {param_exomiser}")
4066
4067        # Param - Assembly
4068        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
4069        log.debug("Assembly: " + str(assembly))
4070
4071        # Data
4072        table_variants = self.get_table_variants()
4073
4074        # Check if not empty
4075        log.debug("Check if not empty")
4076        sql_query_chromosomes = (
4077            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
4078        )
4079        if not self.get_query_to_df(f"{sql_query_chromosomes}")["count"][0]:
4080            log.info(f"VCF empty")
4081            return False
4082
4083        # VCF header
4084        vcf_reader = self.get_header()
4085        log.debug("Initial header: " + str(vcf_reader.infos))
4086
4087        # Samples
4088        samples = self.get_header_sample_list()
4089        if not samples:
4090            log.error("No Samples in VCF")
4091            return False
4092        log.debug(f"Samples: {samples}")
4093
4094        # Memory limit
4095        memory_limit = self.get_memory("8G")
4096        log.debug(f"memory_limit: {memory_limit}")
4097
4098        # Exomiser java options
4099        exomiser_java_options = (
4100            f" -Xmx{memory_limit} -XX:+UseParallelGC -XX:ParallelGCThreads={threads} "
4101        )
4102        log.debug(f"Exomiser java options: {exomiser_java_options}")
4103
4104        # Download Exomiser (if not exists)
4105        exomiser_release = param_exomiser.get("release", None)
4106        exomiser_application_properties = param_exomiser.get(
4107            "exomiser_application_properties", None
4108        )
4109        databases_download_exomiser(
4110            assemblies=[assembly],
4111            exomiser_folder=databases_folders,
4112            exomiser_release=exomiser_release,
4113            exomiser_phenotype_release=exomiser_release,
4114            exomiser_application_properties=exomiser_application_properties,
4115        )
4116
4117        # Force annotation
4118        force_update_annotation = True
4119
4120        if "Exomiser" not in self.get_header().infos or force_update_annotation:
4121            log.debug("Start annotation Exomiser")
4122
4123            with TemporaryDirectory(dir=self.get_tmp_dir()) as tmp_dir:
4124
4125                # tmp_dir = "/tmp/exomiser"
4126
4127                ### ANALYSIS ###
4128                ################
4129
4130                # Create analysis.json through analysis dict
4131                # either analysis in param or by default
4132                # depending on preset exome/genome)
4133
4134                # Init analysis dict
4135                param_exomiser_analysis_dict = {}
4136
4137                # analysis from param
4138                param_exomiser_analysis = param_exomiser.get("analysis", {})
4139                param_exomiser_analysis = full_path(param_exomiser_analysis)
4140
4141                # If analysis in param -> load anlaysis json
4142                if param_exomiser_analysis:
4143
4144                    # If param analysis is a file and exists
4145                    if isinstance(param_exomiser_analysis, str) and os.path.exists(
4146                        param_exomiser_analysis
4147                    ):
4148                        # Load analysis file into analysis dict (either yaml or json)
4149                        with open(param_exomiser_analysis) as json_file:
4150                            param_exomiser_analysis_dict = yaml.safe_load(json_file)
4151
4152                    # If param analysis is a dict
4153                    elif isinstance(param_exomiser_analysis, dict):
4154                        # Load analysis dict into analysis dict (either yaml or json)
4155                        param_exomiser_analysis_dict = param_exomiser_analysis
4156
4157                    # Error analysis type
4158                    else:
4159                        log.error(f"Analysis type unknown. Check param file.")
4160                        raise ValueError(f"Analysis type unknown. Check param file.")
4161
4162                # Case no input analysis config file/dict
4163                # Use preset (exome/genome) to open default config file
4164                if not param_exomiser_analysis_dict:
4165
4166                    # default preset
4167                    default_preset = "exome"
4168
4169                    # Get param preset or default preset
4170                    param_exomiser_preset = param_exomiser.get("preset", default_preset)
4171
4172                    # Try to find if preset is a file
4173                    if os.path.exists(param_exomiser_preset):
4174                        # Preset file is provided in full path
4175                        param_exomiser_analysis_default_config_file = (
4176                            param_exomiser_preset
4177                        )
4178                    # elif os.path.exists(full_path(param_exomiser_preset)):
4179                    #     # Preset file is provided in full path
4180                    #     param_exomiser_analysis_default_config_file = full_path(param_exomiser_preset)
4181                    elif os.path.exists(
4182                        os.path.join(folder_config, param_exomiser_preset)
4183                    ):
4184                        # Preset file is provided a basename in config folder (can be a path with subfolders)
4185                        param_exomiser_analysis_default_config_file = os.path.join(
4186                            folder_config, param_exomiser_preset
4187                        )
4188                    else:
4189                        # Construct preset file
4190                        param_exomiser_analysis_default_config_file = os.path.join(
4191                            folder_config,
4192                            f"preset-{param_exomiser_preset}-analysis.json",
4193                        )
4194
4195                    # If preset file exists
4196                    param_exomiser_analysis_default_config_file = full_path(
4197                        param_exomiser_analysis_default_config_file
4198                    )
4199                    if os.path.exists(param_exomiser_analysis_default_config_file):
4200                        # Load prest file into analysis dict (either yaml or json)
4201                        with open(
4202                            param_exomiser_analysis_default_config_file
4203                        ) as json_file:
4204                            # param_exomiser_analysis_dict[""] = json.load(json_file)
4205                            param_exomiser_analysis_dict["analysis"] = yaml.safe_load(
4206                                json_file
4207                            )
4208
4209                    # Error preset file
4210                    else:
4211                        log.error(
4212                            f"No analysis preset config file ({param_exomiser_analysis_default_config_file})"
4213                        )
4214                        raise ValueError(
4215                            f"No analysis preset config file ({param_exomiser_analysis_default_config_file})"
4216                        )
4217
4218                # If no analysis dict created
4219                if not param_exomiser_analysis_dict:
4220                    log.error(f"No analysis config")
4221                    raise ValueError(f"No analysis config")
4222
4223                # Log
4224                log.debug(f"Pre analysis dict: {param_exomiser_analysis_dict}")
4225
4226                ### PHENOPACKET ###
4227                ###################
4228
4229                # If no PhenoPacket in analysis dict -> check in param
4230                if "phenopacket" not in param_exomiser_analysis_dict:
4231
4232                    # If PhenoPacket in param -> load anlaysis json
4233                    if param_exomiser.get("phenopacket", None):
4234
4235                        param_exomiser_phenopacket = param_exomiser.get("phenopacket")
4236                        param_exomiser_phenopacket = full_path(
4237                            param_exomiser_phenopacket
4238                        )
4239
4240                        # If param phenopacket is a file and exists
4241                        if isinstance(
4242                            param_exomiser_phenopacket, str
4243                        ) and os.path.exists(param_exomiser_phenopacket):
4244                            # Load phenopacket file into analysis dict (either yaml or json)
4245                            with open(param_exomiser_phenopacket) as json_file:
4246                                param_exomiser_analysis_dict["phenopacket"] = (
4247                                    yaml.safe_load(json_file)
4248                                )
4249
4250                        # If param phenopacket is a dict
4251                        elif isinstance(param_exomiser_phenopacket, dict):
4252                            # Load phenopacket dict into analysis dict (either yaml or json)
4253                            param_exomiser_analysis_dict["phenopacket"] = (
4254                                param_exomiser_phenopacket
4255                            )
4256
4257                        # Error phenopacket type
4258                        else:
4259                            log.error(f"Phenopacket type unknown. Check param file.")
4260                            raise ValueError(
4261                                f"Phenopacket type unknown. Check param file."
4262                            )
4263
4264                # If no PhenoPacket in analysis dict -> construct from sample and HPO in param
4265                if "phenopacket" not in param_exomiser_analysis_dict:
4266
4267                    # Init PhenoPacket
4268                    param_exomiser_analysis_dict["phenopacket"] = {
4269                        "id": "analysis",
4270                        "proband": {},
4271                    }
4272
4273                    ### Add subject ###
4274
4275                    # If subject exists
4276                    param_exomiser_subject = param_exomiser.get("subject", {})
4277
4278                    # If subject not exists -> found sample ID
4279                    if not param_exomiser_subject:
4280
4281                        # Found sample ID in param
4282                        sample = param_exomiser.get("sample", None)
4283
4284                        # Find sample ID (first sample)
4285                        if not sample:
4286                            sample_list = self.get_header_sample_list()
4287                            if len(sample_list) > 0:
4288                                sample = sample_list[0]
4289                            else:
4290                                log.error(f"No sample found")
4291                                raise ValueError(f"No sample found")
4292
4293                        # Create subject
4294                        param_exomiser_subject = {"id": sample, "sex": "UNKNOWN_SEX"}
4295
4296                    # Add to dict
4297                    param_exomiser_analysis_dict["phenopacket"][
4298                        "subject"
4299                    ] = param_exomiser_subject
4300
4301                    ### Add "phenotypicFeatures" ###
4302
4303                    # If phenotypicFeatures exists
4304                    param_exomiser_phenotypicfeatures = param_exomiser.get(
4305                        "phenotypicFeatures", []
4306                    )
4307
4308                    # If phenotypicFeatures not exists -> Try to infer from hpo list
4309                    if not param_exomiser_phenotypicfeatures:
4310
4311                        # Found HPO in param
4312                        param_exomiser_hpo = param_exomiser.get("hpo", [])
4313
4314                        # Split HPO if list in string format separated by comma
4315                        if isinstance(param_exomiser_hpo, str):
4316                            param_exomiser_hpo = param_exomiser_hpo.split(",")
4317
4318                        # Create HPO list
4319                        for hpo in param_exomiser_hpo:
4320                            hpo_clean = re.sub("[^0-9]", "", hpo)
4321                            param_exomiser_phenotypicfeatures.append(
4322                                {
4323                                    "type": {
4324                                        "id": f"HP:{hpo_clean}",
4325                                        "label": f"HP:{hpo_clean}",
4326                                    }
4327                                }
4328                            )
4329
4330                    # Add to dict
4331                    param_exomiser_analysis_dict["phenopacket"][
4332                        "phenotypicFeatures"
4333                    ] = param_exomiser_phenotypicfeatures
4334
4335                    # If phenotypicFeatures not exists -> Remove hiPhivePrioritiser step
4336                    if not param_exomiser_phenotypicfeatures:
4337                        for step in param_exomiser_analysis_dict.get(
4338                            "analysis", {}
4339                        ).get("steps", []):
4340                            if "hiPhivePrioritiser" in step:
4341                                param_exomiser_analysis_dict.get("analysis", {}).get(
4342                                    "steps", []
4343                                ).remove(step)
4344
4345                ### Add Input File ###
4346
4347                # Initial file name and htsFiles
4348                tmp_vcf_name = os.path.join(tmp_dir, "initial.vcf.gz")
4349                param_exomiser_analysis_dict["phenopacket"]["htsFiles"] = [
4350                    {
4351                        "uri": tmp_vcf_name,
4352                        "htsFormat": "VCF",
4353                        "genomeAssembly": assembly,
4354                    }
4355                ]
4356
4357                ### Add metaData ###
4358
4359                # If metaData not in analysis dict
4360                if "metaData" not in param_exomiser_analysis_dict:
4361                    param_exomiser_analysis_dict["phenopacket"]["metaData"] = {
4362                        "created": f"{datetime.datetime.now()}".replace(" ", "T") + "Z",
4363                        "createdBy": "howard",
4364                        "phenopacketSchemaVersion": 1,
4365                    }
4366
4367                ### OutputOptions ###
4368
4369                # Init output result folder
4370                output_results = os.path.join(tmp_dir, "results")
4371
4372                # If no outputOptions in analysis dict
4373                if "outputOptions" not in param_exomiser_analysis_dict:
4374
4375                    # default output formats
4376                    defaut_output_formats = ["TSV_VARIANT", "VCF"]
4377
4378                    # Get outputOptions in param
4379                    output_options = param_exomiser.get("outputOptions", None)
4380
4381                    # If no output_options in param -> check
4382                    if not output_options:
4383                        output_options = {
4384                            "outputContributingVariantsOnly": False,
4385                            "numGenes": 0,
4386                            "outputFormats": defaut_output_formats,
4387                        }
4388
4389                    # Replace outputDirectory in output options
4390                    output_options["outputDirectory"] = output_results
4391                    output_options["outputFileName"] = "howard"
4392
4393                    # Add outputOptions in analysis dict
4394                    param_exomiser_analysis_dict["outputOptions"] = output_options
4395
4396                else:
4397
4398                    # Replace output_results and output format (if exists in param)
4399                    param_exomiser_analysis_dict["outputOptions"][
4400                        "outputDirectory"
4401                    ] = output_results
4402                    param_exomiser_analysis_dict["outputOptions"]["outputFormats"] = (
4403                        list(
4404                            set(
4405                                param_exomiser_analysis_dict.get(
4406                                    "outputOptions", {}
4407                                ).get("outputFormats", [])
4408                                + ["TSV_VARIANT", "VCF"]
4409                            )
4410                        )
4411                    )
4412
4413                # log
4414                log.debug(f"Pre analysis dict: {param_exomiser_analysis_dict}")
4415
4416                ### ANALYSIS FILE ###
4417                #####################
4418
4419                ### Full JSON analysis config file ###
4420
4421                exomiser_analysis = os.path.join(tmp_dir, "analysis.json")
4422                with open(exomiser_analysis, "w") as fp:
4423                    json.dump(param_exomiser_analysis_dict, fp, indent=4)
4424
4425                ### SPLIT analysis and sample config files
4426
4427                # Splitted analysis dict
4428                param_exomiser_analysis_dict_for_split = (
4429                    param_exomiser_analysis_dict.copy()
4430                )
4431
4432                # Phenopacket JSON file
4433                exomiser_analysis_phenopacket = os.path.join(
4434                    tmp_dir, "analysis_phenopacket.json"
4435                )
4436                with open(exomiser_analysis_phenopacket, "w") as fp:
4437                    json.dump(
4438                        param_exomiser_analysis_dict_for_split.get("phenopacket"),
4439                        fp,
4440                        indent=4,
4441                    )
4442
4443                # Analysis JSON file without Phenopacket parameters
4444                param_exomiser_analysis_dict_for_split.pop("phenopacket")
4445                exomiser_analysis_analysis = os.path.join(
4446                    tmp_dir, "analysis_analysis.json"
4447                )
4448                with open(exomiser_analysis_analysis, "w") as fp:
4449                    json.dump(param_exomiser_analysis_dict_for_split, fp, indent=4)
4450
4451                ### INITAL VCF file ###
4452                #######################
4453
4454                ### Create list of samples to use and include inti initial VCF file ####
4455
4456                # Subject (main sample)
4457                # Get sample ID in analysis dict
4458                sample_subject = (
4459                    param_exomiser_analysis_dict.get("phenopacket", {})
4460                    .get("subject", {})
4461                    .get("id", None)
4462                )
4463                sample_proband = (
4464                    param_exomiser_analysis_dict.get("phenopacket", {})
4465                    .get("proband", {})
4466                    .get("subject", {})
4467                    .get("id", None)
4468                )
4469                sample = []
4470                if sample_subject:
4471                    sample.append(sample_subject)
4472                if sample_proband:
4473                    sample.append(sample_proband)
4474
4475                # Get sample ID within Pedigree
4476                pedigree_persons_list = (
4477                    param_exomiser_analysis_dict.get("phenopacket", {})
4478                    .get("pedigree", {})
4479                    .get("persons", {})
4480                )
4481
4482                # Create list with all sample ID in pedigree (if exists)
4483                pedigree_persons = []
4484                for person in pedigree_persons_list:
4485                    pedigree_persons.append(person.get("individualId"))
4486
4487                # Concat subject sample ID and samples ID in pedigreesamples
4488                samples = list(set(sample + pedigree_persons))
4489
4490                # Check if sample list is not empty
4491                if not samples:
4492                    log.error(f"No samples found")
4493                    raise ValueError(f"No samples found")
4494
4495                # Create VCF with sample (either sample in param or first one by default)
4496                # Export VCF file
4497                self.export_variant_vcf(
4498                    vcf_file=tmp_vcf_name,
4499                    remove_info=True,
4500                    add_samples=True,
4501                    list_samples=samples,
4502                    index=False,
4503                )
4504
4505                ### Execute Exomiser ###
4506                ########################
4507
4508                # Init command
4509                exomiser_command = ""
4510
4511                # Command exomiser options
4512                exomiser_options = f" --spring.config.location={databases_folders}/{assembly}/application.properties --exomiser.data-directory={databases_folders}/{assembly} "
4513
4514                # Release
4515                exomiser_release = param_exomiser.get("release", None)
4516                if exomiser_release:
4517                    # phenotype data version
4518                    exomiser_options += (
4519                        f" --exomiser.phenotype.data-version={exomiser_release} "
4520                    )
4521                    # data version
4522                    exomiser_options += (
4523                        f" --exomiser.{assembly}.data-version={exomiser_release} "
4524                    )
4525                    # variant white list
4526                    variant_white_list_file = (
4527                        f"{exomiser_release}_{assembly}_clinvar_whitelist.tsv.gz"
4528                    )
4529                    if os.path.exists(
4530                        os.path.join(
4531                            databases_folders, assembly, variant_white_list_file
4532                        )
4533                    ):
4534                        exomiser_options += f" --exomiser.{assembly}.variant-white-list-path={variant_white_list_file} "
4535
4536                # transcript_source
4537                transcript_source = param_exomiser.get(
4538                    "transcript_source", None
4539                )  # ucsc, refseq, ensembl
4540                if transcript_source:
4541                    exomiser_options += (
4542                        f" --exomiser.{assembly}.transcript-source={transcript_source} "
4543                    )
4544
4545                # If analysis contain proband param
4546                if param_exomiser_analysis_dict.get("phenopacket", {}).get(
4547                    "proband", {}
4548                ):
4549                    exomiser_command_analysis = f" {exomiser_bin_command} --analysis={exomiser_analysis_analysis} --sample={exomiser_analysis_phenopacket} {exomiser_options} "
4550
4551                # If no proband (usually uniq sample)
4552                else:
4553                    exomiser_command_analysis = f" {exomiser_bin_command} --analysis={exomiser_analysis} {exomiser_options}"
4554
4555                # Log
4556                log.debug(f"exomiser_command_analysis={exomiser_command_analysis}")
4557
4558                # Run command
4559                result = subprocess.call(
4560                    exomiser_command_analysis.split(), stdout=subprocess.PIPE
4561                )
4562                if result:
4563                    log.error("Exomiser command failed")
4564                    raise ValueError("Exomiser command failed")
4565
4566                ### RESULTS ###
4567                ###############
4568
4569                ### Annotate with TSV fields ###
4570
4571                # Init result tsv file
4572                exomiser_to_info = param_exomiser.get("exomiser_to_info", False)
4573
4574                # Init result tsv file
4575                output_results_tsv = os.path.join(output_results, "howard.variants.tsv")
4576
4577                # Parse TSV file and explode columns in INFO field
4578                if exomiser_to_info and os.path.exists(output_results_tsv):
4579
4580                    # Log
4581                    log.debug("Exomiser columns to VCF INFO field")
4582
4583                    # Retrieve columns and types
4584                    query = f""" SELECT * FROM read_csv('{output_results_tsv}', auto_detect=True, delim='\t', sample_size=-1) LIMIT 0 """
4585                    output_results_tsv_df = self.get_query_to_df(query)
4586                    output_results_tsv_columns = output_results_tsv_df.columns.tolist()
4587
4588                    # Init concat fields for update
4589                    sql_query_update_concat_fields = []
4590
4591                    # Fields to avoid
4592                    fields_to_avoid = [
4593                        "CONTIG",
4594                        "START",
4595                        "END",
4596                        "REF",
4597                        "ALT",
4598                        "QUAL",
4599                        "FILTER",
4600                        "GENOTYPE",
4601                    ]
4602
4603                    # List all columns to add into header
4604                    for header_column in output_results_tsv_columns:
4605
4606                        # If header column is enable
4607                        if header_column not in fields_to_avoid:
4608
4609                            # Header info type
4610                            header_info_type = "String"
4611                            header_column_df = output_results_tsv_df[header_column]
4612                            header_column_df_dtype = header_column_df.dtype
4613                            if header_column_df_dtype == object:
4614                                if (
4615                                    pd.to_numeric(header_column_df, errors="coerce")
4616                                    .notnull()
4617                                    .all()
4618                                ):
4619                                    header_info_type = "Float"
4620                            else:
4621                                header_info_type = "Integer"
4622
4623                            # Header info
4624                            characters_to_validate = ["-"]
4625                            pattern = "[" + "".join(characters_to_validate) + "]"
4626                            header_info_name = re.sub(
4627                                pattern,
4628                                "_",
4629                                f"Exomiser_{header_column}".replace("#", ""),
4630                            )
4631                            header_info_number = "."
4632                            header_info_description = (
4633                                f"Exomiser {header_column} annotation"
4634                            )
4635                            header_info_source = "Exomiser"
4636                            header_info_version = "unknown"
4637                            header_info_code = CODE_TYPE_MAP[header_info_type]
4638                            vcf_reader.infos[header_info_name] = vcf.parser._Info(
4639                                header_info_name,
4640                                header_info_number,
4641                                header_info_type,
4642                                header_info_description,
4643                                header_info_source,
4644                                header_info_version,
4645                                header_info_code,
4646                            )
4647
4648                            # Add field to add for update to concat fields
4649                            sql_query_update_concat_fields.append(
4650                                f"""
4651                                CASE
4652                                    WHEN table_parquet."{header_column}" NOT IN ('','.')
4653                                    THEN concat(
4654                                        '{header_info_name}=',
4655                                        table_parquet."{header_column}",
4656                                        ';'
4657                                        )
4658
4659                                    ELSE ''
4660                                END
4661                            """
4662                            )
4663
4664                    # Update query
4665                    sql_query_update = f"""
4666                        UPDATE {table_variants} as table_variants
4667                            SET INFO = concat(
4668                                            CASE
4669                                                WHEN INFO NOT IN ('', '.')
4670                                                THEN INFO
4671                                                ELSE ''
4672                                            END,
4673                                            CASE
4674                                                WHEN table_variants.INFO NOT IN ('','.')
4675                                                THEN ';'
4676                                                ELSE ''
4677                                            END,
4678                                            (
4679                                            SELECT 
4680                                                concat(
4681                                                    {",".join(sql_query_update_concat_fields)}
4682                                                )
4683                                            FROM read_csv('{output_results_tsv}', auto_detect=True, delim='\t', sample_size=-1) as table_parquet
4684                                                    WHERE concat('chr', CAST(table_parquet.\"CONTIG\" AS STRING)) = table_variants.\"#CHROM\"
4685                                                    AND table_parquet.\"START\" = table_variants.\"POS\"
4686                                                    AND table_parquet.\"ALT\" = table_variants.\"ALT\"
4687                                                    AND table_parquet.\"REF\" = table_variants.\"REF\"
4688                                            )
4689                                        )
4690                            ;
4691                        """
4692
4693                    # Update
4694                    self.conn.execute(sql_query_update)
4695
4696                ### Annotate with VCF INFO field ###
4697
4698                # Init result VCF file
4699                output_results_vcf = os.path.join(output_results, "howard.vcf.gz")
4700
4701                # If VCF exists
4702                if os.path.exists(output_results_vcf):
4703
4704                    # Log
4705                    log.debug("Exomiser result VCF update variants")
4706
4707                    # Find Exomiser INFO field annotation in header
4708                    with gzip.open(output_results_vcf, "rt") as f:
4709                        header_list = self.read_vcf_header(f)
4710                    exomiser_vcf_header = vcf.Reader(
4711                        io.StringIO("\n".join(header_list))
4712                    )
4713
4714                    # Add annotation INFO field to header
4715                    vcf_reader.infos["Exomiser"] = exomiser_vcf_header.infos["Exomiser"]
4716
4717                    # Update variants with VCF
4718                    self.update_from_vcf(output_results_vcf)
4719
4720        return True

This function annotate with Exomiser

This function uses args as parameters, in section "annotation" -> "exomiser", with sections:

"analysis" (dict/file): Full analysis dictionnary parameters (see Exomiser docs). Either a dict, or a file in JSON or YAML format. These parameters may change depending on other parameters (e.g. phenotipicFeatures/HPO) Default : None
"preset" (string): Analysis preset (available in config folder). Used if no full "analysis" is provided. Default: "exome"
"phenopacket" (dict/file): Samples and phenotipic features parameters (see Exomiser docs). Either a dict, or a file in JSON or YAML format. Default: None
"subject" (dict): Sample parameters (see Exomiser docs). Example: "subject": { "id": "ISDBM322017", "sex": "FEMALE" } Default: None
"sample" (string): Sample name to construct "subject" section: "subject": { "id": "", "sex": "UNKNOWN_SEX" } Default: None
"phenotypicFeatures" (dict) Phenotypic features to construct "subject" section. Example: "phenotypicFeatures": [ { "type": { "id": "HP:0001159", "label": "Syndactyly" } }, { "type": { "id": "HP:0000486", "label": "Strabismus" } } ]
"hpo" (list) List of HPO ids as phenotypic features. Example: "hpo": ['0001156', '0001363', '0011304', '0010055'] Default: []
"outputOptions" (dict): Output options (see Exomiser docs). Default: "output_options" = { "outputContributingVariantsOnly": False, "numGenes": 0, "outputFormats": ["TSV_VARIANT", "VCF"] }
"transcript_source" (string): Transcript source (either "refseq", "ucsc", "ensembl") Default: "refseq"
"exomiser_to_info" (boolean): Add exomiser TSV file columns as INFO fields in VCF. Default: False
"release" (string): Exomise database release. If not exists, database release will be downloaded (take a while). Default: None (provided by application.properties configuration file)
"exomiser_application_properties" (file): Exomiser configuration file (see Exomiser docs). Useful to automatically download databases (especially for specific genome databases).

Notes:

If no sample in parameters, first sample in VCF will be chosen
If no HPO found, "hiPhivePrioritiser" analysis step will be switch off

Parameters

threads: The number of threads to use

Returns

None.

def annotation_snpeff(self, threads: int = None) -> None: View Source

4722    def annotation_snpeff(self, threads: int = None) -> None:
4723        """
4724        This function annotate with snpEff
4725
4726        :param threads: The number of threads to use
4727        :return: the value of the variable "return_value".
4728        """
4729
4730        # DEBUG
4731        log.debug("Start annotation with snpeff databases")
4732
4733        # Threads
4734        if not threads:
4735            threads = self.get_threads()
4736        log.debug("Threads: " + str(threads))
4737
4738        # DEBUG
4739        delete_tmp = True
4740        if self.get_config().get("verbosity", "warning") in ["debug"]:
4741            delete_tmp = False
4742            log.debug("Delete tmp files/folders: " + str(delete_tmp))
4743
4744        # Config
4745        config = self.get_config()
4746        log.debug("Config: " + str(config))
4747
4748        # Config - Folders - Databases
4749        databases_folders = (
4750            config.get("folders", {}).get("databases", {}).get("snpeff", ["."])
4751        )
4752        log.debug("Databases annotations: " + str(databases_folders))
4753
4754        # # Config - Java
4755        # java_bin = get_bin(
4756        #     tool="java",
4757        #     bin="java",
4758        #     bin_type="bin",
4759        #     config=config,
4760        #     default_folder="/usr/bin",
4761        # )
4762        # if not (os.path.exists(java_bin) or (java_bin and which(java_bin))):
4763        #     log.error(f"Annotation failed: no java bin '{java_bin}'")
4764        #     raise ValueError(f"Annotation failed: no java bin '{java_bin}'")
4765
4766        # # Config - snpEff bin
4767        # snpeff_jar = get_bin(
4768        #     tool="snpeff",
4769        #     bin="snpEff.jar",
4770        #     bin_type="jar",
4771        #     config=config,
4772        #     default_folder=f"{DEFAULT_TOOLS_FOLDER}/snpeff",
4773        # )
4774        # if not (os.path.exists(snpeff_jar) or (snpeff_jar and which(snpeff_jar))):
4775        #     log.error(f"Annotation failed: no snpEff jar '{snpeff_jar}'")
4776        #     raise ValueError(f"Annotation failed: no snpEff jar '{snpeff_jar}'")
4777
4778        # Config - snpEff bin command
4779        snpeff_bin_command = get_bin_command(
4780            bin="snpEff.jar",
4781            tool="snpeff",
4782            bin_type="jar",
4783            config=config,
4784            default_folder=f"{DEFAULT_TOOLS_FOLDER}/snpeff",
4785        )
4786        if not snpeff_bin_command:
4787            msg_err = f"Annotation failed: no snpeff bin '{snpeff_bin_command}'"
4788            log.error(msg_err)
4789            raise ValueError(msg_err)
4790
4791        # Config - snpEff databases
4792        snpeff_databases = (
4793            config.get("folders", {})
4794            .get("databases", {})
4795            .get("snpeff", DEFAULT_SNPEFF_FOLDER)
4796        )
4797        snpeff_databases = full_path(snpeff_databases)
4798        if snpeff_databases is not None and snpeff_databases != "":
4799            log.debug(f"Create snpEff databases folder")
4800            if not os.path.exists(snpeff_databases):
4801                os.makedirs(snpeff_databases)
4802
4803        # Param
4804        param = self.get_param()
4805        log.debug("Param: " + str(param))
4806
4807        # Param
4808        options = param.get("annotation", {}).get("snpeff", {}).get("options", None)
4809        log.debug("Options: " + str(options))
4810
4811        # Param - Assembly
4812        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
4813
4814        # Param - Options
4815        snpeff_options = (
4816            param.get("annotation", {}).get("snpeff", {}).get("options", "")
4817        )
4818        snpeff_stats = param.get("annotation", {}).get("snpeff", {}).get("stats", None)
4819        snpeff_csvstats = (
4820            param.get("annotation", {}).get("snpeff", {}).get("csvStats", None)
4821        )
4822        if snpeff_stats:
4823            snpeff_stats = snpeff_stats.replace("OUTPUT", self.get_output())
4824            snpeff_stats = full_path(snpeff_stats)
4825            snpeff_options += f" -stats {snpeff_stats}"
4826        if snpeff_csvstats:
4827            snpeff_csvstats = snpeff_csvstats.replace("OUTPUT", self.get_output())
4828            snpeff_csvstats = full_path(snpeff_csvstats)
4829            snpeff_options += f" -csvStats {snpeff_csvstats}"
4830
4831        # Data
4832        table_variants = self.get_table_variants()
4833
4834        # Check if not empty
4835        log.debug("Check if not empty")
4836        sql_query_chromosomes = (
4837            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
4838        )
4839        # if not self.conn.execute(f"{sql_query_chromosomes}").df()["count"][0]:
4840        if not self.get_query_to_df(f"{sql_query_chromosomes}")["count"][0]:
4841            log.info(f"VCF empty")
4842            return
4843
4844        # Export in VCF
4845        log.debug("Create initial file to annotate")
4846        tmp_vcf = NamedTemporaryFile(
4847            prefix=self.get_prefix(),
4848            dir=self.get_tmp_dir(),
4849            suffix=".vcf.gz",
4850            delete=True,
4851        )
4852        tmp_vcf_name = tmp_vcf.name
4853
4854        # VCF header
4855        vcf_reader = self.get_header()
4856        log.debug("Initial header: " + str(vcf_reader.infos))
4857
4858        # Existing annotations
4859        for vcf_annotation in self.get_header().infos:
4860
4861            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
4862            log.debug(
4863                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
4864            )
4865
4866        # Memory limit
4867        # if config.get("memory", None):
4868        #     memory_limit = config.get("memory", "8G")
4869        # else:
4870        #     memory_limit = "8G"
4871        memory_limit = self.get_memory("8G")
4872        log.debug(f"memory_limit: {memory_limit}")
4873
4874        # snpEff java options
4875        snpeff_java_options = (
4876            f" -Xmx{memory_limit} -XX:+UseParallelGC -XX:ParallelGCThreads={threads} "
4877        )
4878        log.debug(f"Exomiser java options: {snpeff_java_options}")
4879
4880        force_update_annotation = True
4881
4882        if "ANN" not in self.get_header().infos or force_update_annotation:
4883
4884            # Check snpEff database
4885            log.debug(f"Check snpEff databases {[assembly]}")
4886            databases_download_snpeff(
4887                folder=snpeff_databases, assemblies=[assembly], config=config
4888            )
4889
4890            # Export VCF file
4891            self.export_variant_vcf(
4892                vcf_file=tmp_vcf_name,
4893                remove_info=True,
4894                add_samples=False,
4895                index=True,
4896            )
4897
4898            # Tmp file
4899            err_files = []
4900            tmp_annotate_vcf = NamedTemporaryFile(
4901                prefix=self.get_prefix(),
4902                dir=self.get_tmp_dir(),
4903                suffix=".vcf",
4904                delete=False,
4905            )
4906            tmp_annotate_vcf_name = tmp_annotate_vcf.name
4907            tmp_annotate_vcf_name_err = tmp_annotate_vcf_name + ".err"
4908            err_files.append(tmp_annotate_vcf_name_err)
4909
4910            # Command
4911            snpeff_command = f"{snpeff_bin_command} {assembly} -dataDir {snpeff_databases} {snpeff_options} {tmp_vcf_name} 1>{tmp_annotate_vcf_name} 2>>{tmp_annotate_vcf_name_err}"
4912            log.debug(f"Annotation - snpEff command: {snpeff_command}")
4913            run_parallel_commands([snpeff_command], 1)
4914
4915            # Error messages
4916            log.info(f"Error/Warning messages:")
4917            error_message_command_all = []
4918            error_message_command_warning = []
4919            error_message_command_err = []
4920            for err_file in err_files:
4921                with open(err_file, "r") as f:
4922                    for line in f:
4923                        message = line.strip()
4924                        error_message_command_all.append(message)
4925                        if line.startswith("[W::"):
4926                            error_message_command_warning.append(message)
4927                        if line.startswith("[E::"):
4928                            error_message_command_err.append(f"{err_file}: " + message)
4929            # log info
4930            for message in list(
4931                set(error_message_command_err + error_message_command_warning)
4932            ):
4933                log.info(f"   {message}")
4934            # debug info
4935            for message in list(set(error_message_command_all)):
4936                log.debug(f"   {message}")
4937            # failed
4938            if len(error_message_command_err):
4939                log.error("Annotation failed: Error in commands")
4940                raise ValueError("Annotation failed: Error in commands")
4941
4942            # Find annotation in header
4943            with open(tmp_annotate_vcf_name, "rt") as f:
4944                header_list = self.read_vcf_header(f)
4945            annovar_vcf_header = vcf.Reader(io.StringIO("\n".join(header_list)))
4946
4947            for ann in annovar_vcf_header.infos:
4948                if ann not in self.get_header().infos:
4949                    vcf_reader.infos[ann] = annovar_vcf_header.infos.get(ann)
4950
4951            # Update variants
4952            log.info(f"Annotation - Updating...")
4953            self.update_from_vcf(tmp_annotate_vcf_name)
4954
4955        else:
4956            if "ANN" in self.get_header().infos:
4957                log.debug(f"Existing snpEff annotations in VCF")
4958            if force_update_annotation:
4959                log.debug(f"Existing snpEff annotations in VCF - annotation forced")

This function annotate with snpEff

Parameters

threads: The number of threads to use

Returns

the value of the variable "return_value".

def annotation_annovar(self, threads: int = None) -> None: View Source

4961    def annotation_annovar(self, threads: int = None) -> None:
4962        """
4963        It takes a VCF file, annotates it with Annovar, and then updates the database with the new
4964        annotations
4965
4966        :param threads: number of threads to use
4967        :return: the value of the variable "return_value".
4968        """
4969
4970        # DEBUG
4971        log.debug("Start annotation with Annovar databases")
4972
4973        # Threads
4974        if not threads:
4975            threads = self.get_threads()
4976        log.debug("Threads: " + str(threads))
4977
4978        # Tmp en Err files
4979        tmp_files = []
4980        err_files = []
4981
4982        # DEBUG
4983        delete_tmp = True
4984        if self.get_config().get("verbosity", "warning") in ["debug"]:
4985            delete_tmp = False
4986            log.debug("Delete tmp files/folders: " + str(delete_tmp))
4987
4988        # Config
4989        config = self.get_config()
4990        log.debug("Config: " + str(config))
4991
4992        # Config - Folders - Databases
4993        databases_folders = (
4994            config.get("folders", {}).get("databases", {}).get("annovar", ["."])
4995        )
4996        log.debug("Databases annotations: " + str(databases_folders))
4997
4998        # Config - annovar bin command
4999        annovar_bin_command = get_bin_command(
5000            bin="table_annovar.pl",
5001            tool="annovar",
5002            bin_type="perl",
5003            config=config,
5004            default_folder=f"{DEFAULT_TOOLS_FOLDER}/annovar",
5005        )
5006        if not annovar_bin_command:
5007            msg_err = f"Annotation failed: no annovar bin '{annovar_bin_command}'"
5008            log.error(msg_err)
5009            raise ValueError(msg_err)
5010
5011        # Config - BCFTools bin command
5012        bcftools_bin_command = get_bin_command(
5013            bin="bcftools",
5014            tool="bcftools",
5015            bin_type="bin",
5016            config=config,
5017            default_folder=f"{DEFAULT_TOOLS_FOLDER}/bcftools",
5018        )
5019        if not bcftools_bin_command:
5020            msg_err = f"Annotation failed: no bcftools bin '{bcftools_bin_command}'"
5021            log.error(msg_err)
5022            raise ValueError(msg_err)
5023
5024        # Config - annovar databases
5025        annovar_databases = (
5026            config.get("folders", {})
5027            .get("databases", {})
5028            .get("annovar", DEFAULT_ANNOVAR_FOLDER)
5029        )
5030        annovar_databases = full_path(annovar_databases)
5031        if annovar_databases != "" and not os.path.exists(annovar_databases):
5032            os.makedirs(annovar_databases)
5033
5034        # Param
5035        param = self.get_param()
5036        log.debug("Param: " + str(param))
5037
5038        # Param - options
5039        options = param.get("annotation", {}).get("annovar", {}).get("options", {})
5040        log.debug("Options: " + str(options))
5041
5042        # Param - annotations
5043        annotations = (
5044            param.get("annotation", {}).get("annovar", {}).get("annotations", {})
5045        )
5046        log.debug("Annotations: " + str(annotations))
5047
5048        # Param - Assembly
5049        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
5050
5051        # Annovar database assembly
5052        annovar_databases_assembly = f"{annovar_databases}/{assembly}"
5053        if annovar_databases_assembly != "" and not os.path.exists(
5054            annovar_databases_assembly
5055        ):
5056            os.makedirs(annovar_databases_assembly)
5057
5058        # Data
5059        table_variants = self.get_table_variants()
5060
5061        # Check if not empty
5062        log.debug("Check if not empty")
5063        sql_query_chromosomes = (
5064            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
5065        )
5066        sql_query_chromosomes_df = self.get_query_to_df(sql_query_chromosomes)
5067        if not sql_query_chromosomes_df["count"][0]:
5068            log.info(f"VCF empty")
5069            return
5070
5071        # VCF header
5072        vcf_reader = self.get_header()
5073        log.debug("Initial header: " + str(vcf_reader.infos))
5074
5075        # Existing annotations
5076        for vcf_annotation in self.get_header().infos:
5077
5078            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
5079            log.debug(
5080                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
5081            )
5082
5083        force_update_annotation = True
5084
5085        if annotations:
5086
5087            commands = []
5088            tmp_annotates_vcf_name_list = []
5089
5090            # Export in VCF
5091            log.debug("Create initial file to annotate")
5092            tmp_vcf = NamedTemporaryFile(
5093                prefix=self.get_prefix(),
5094                dir=self.get_tmp_dir(),
5095                suffix=".vcf.gz",
5096                delete=False,
5097            )
5098            tmp_vcf_name = tmp_vcf.name
5099            tmp_files.append(tmp_vcf_name)
5100            tmp_files.append(tmp_vcf_name + ".tbi")
5101
5102            # Export VCF file
5103            self.export_variant_vcf(
5104                vcf_file=tmp_vcf_name,
5105                remove_info=".",
5106                add_samples=False,
5107                index=True,
5108            )
5109
5110            # Create file for field rename
5111            log.debug("Create file for field rename")
5112            tmp_rename = NamedTemporaryFile(
5113                prefix=self.get_prefix(),
5114                dir=self.get_tmp_dir(),
5115                suffix=".rename",
5116                delete=False,
5117            )
5118            tmp_rename_name = tmp_rename.name
5119            tmp_files.append(tmp_rename_name)
5120
5121            # Check Annovar database
5122            log.debug(
5123                f"Check Annovar databases {[assembly]}: {list(annotations.keys())}"
5124            )
5125            databases_download_annovar(
5126                folder=annovar_databases,
5127                files=list(annotations.keys()),
5128                assemblies=[assembly],
5129            )
5130
5131            for annotation in annotations:
5132                annotation_fields = annotations[annotation]
5133
5134                if not annotation_fields:
5135                    annotation_fields = {"INFO": None}
5136
5137                log.info(f"Annotations Annovar - database '{annotation}'")
5138                log.debug(f"Annotation '{annotation}' - fields: {annotation_fields}")
5139
5140                # Tmp file for annovar
5141                err_files = []
5142                tmp_annotate_vcf_directory = TemporaryDirectory(
5143                    prefix=self.get_prefix(), dir=self.get_tmp_dir(), suffix=".annovar"
5144                )
5145                tmp_annotate_vcf_prefix = tmp_annotate_vcf_directory.name + "/annovar"
5146                tmp_annotate_vcf_name_annovar = (
5147                    tmp_annotate_vcf_prefix + "." + assembly + "_multianno.vcf"
5148                )
5149                tmp_annotate_vcf_name_err = tmp_annotate_vcf_directory.name + "/.err"
5150                err_files.append(tmp_annotate_vcf_name_err)
5151                tmp_files.append(tmp_annotate_vcf_name_err)
5152
5153                # Tmp file final vcf annotated by annovar
5154                tmp_annotate_vcf = NamedTemporaryFile(
5155                    prefix=self.get_prefix(),
5156                    dir=self.get_tmp_dir(),
5157                    suffix=".vcf.gz",
5158                    delete=False,
5159                )
5160                tmp_annotate_vcf_name = tmp_annotate_vcf.name
5161                tmp_annotates_vcf_name_list.append(tmp_annotate_vcf_name)
5162                tmp_files.append(tmp_annotate_vcf_name)
5163                tmp_files.append(tmp_annotate_vcf_name + ".tbi")
5164
5165                # Number of fields
5166                annotation_list = []
5167                annotation_renamed_list = []
5168
5169                for annotation_field in annotation_fields:
5170
5171                    # field new name, if parametered SKIPPED !!!!!! not managed actually TODO
5172                    annotation_fields_new_name = annotation_fields.get(
5173                        annotation_field, annotation_field
5174                    )
5175                    if not annotation_fields_new_name:
5176                        annotation_fields_new_name = annotation_field
5177
5178                    if (
5179                        force_update_annotation
5180                        or annotation_fields_new_name not in self.get_header().infos
5181                    ):
5182                        annotation_list.append(annotation_field)
5183                        annotation_renamed_list.append(annotation_fields_new_name)
5184                    else:  # annotation_fields_new_name in self.get_header().infos and not force_update_annotation:
5185                        log.warning(
5186                            f"Annotation '{annotation}' - '{annotation_fields_new_name}' - already exists (skipped)"
5187                        )
5188
5189                    # Add rename info
5190                    run_parallel_commands(
5191                        [
5192                            f"echo 'INFO/{annotation_field} {annotation_fields_new_name}' >> {tmp_rename_name}"
5193                        ],
5194                        1,
5195                    )
5196
5197                # log.debug("fields_to_removed: " + str(fields_to_removed))
5198                log.debug("annotation_list: " + str(annotation_list))
5199
5200                # protocol
5201                protocol = annotation
5202
5203                # argument
5204                argument = ""
5205
5206                # operation
5207                operation = "f"
5208                if annotation in ["refGene", "refGeneWithVer"] or annotation.startswith(
5209                    "ensGene"
5210                ):
5211                    operation = "g"
5212                    if options.get("genebase", None):
5213                        argument = f"""'{options.get("genebase","")}'"""
5214                elif annotation in ["cytoBand"]:
5215                    operation = "r"
5216
5217                # argument option
5218                argument_option = ""
5219                if argument != "":
5220                    argument_option = " --argument " + argument
5221
5222                # command options
5223                command_options = f""" --nastring . --vcfinput --polish --dot2underline --thread {threads} """  # --intronhgvs 10
5224                for option in options:
5225                    if option not in ["genebase"]:
5226                        command_options += f""" --{option}={options[option]}"""
5227
5228                # Command
5229
5230                # Command - Annovar
5231                command_annovar = f"""{annovar_bin_command} {tmp_vcf_name} {annovar_databases_assembly} --buildver {assembly} --outfile {tmp_annotate_vcf_prefix} --remove --protocol {protocol} --operation {operation} {argument_option} {command_options} 2>>{tmp_annotate_vcf_name_err} && mv {tmp_annotate_vcf_name_annovar} {tmp_annotate_vcf_name}.tmp.vcf """
5232                tmp_files.append(f"{tmp_annotate_vcf_name}.tmp.vcf")
5233
5234                # Command - start pipe
5235                command_annovar += f""" && {bcftools_bin_command} view --threads={threads} {tmp_annotate_vcf_name}.tmp.vcf 2>>{tmp_annotate_vcf_name_err} """
5236
5237                # Command - Clean INFO/ANNOVAR_DATE (due to Annovar issue with multiple TAGS!)
5238                command_annovar += """ | sed "s/ANNOVAR_DATE=[^;\t]*;//gi" """
5239
5240                # Command - Special characters (refGene annotation)
5241                command_annovar += """ | sed "s/\\\\\\x3b/,/gi" """
5242
5243                # Command - Clean empty fields (with value ".")
5244                command_annovar += """ | awk -F'\\t' -v OFS='\\t' '{if ($0 ~ /^#/) print; else {split($8,a,";");for(i=1;i<=length(a);i++) {split(a[i],b,"=");if(b[2]!=".") {c[b[1]]=b[2]}}; split($8,d,";");for(i=1;i<=length(d);i++) {split(d[i],e,"=");if(c[e[1]]!="") {if(info!="") {info=info";"}; info=info""e[1]"="c[e[1]]}}; if(info!="") {$8=info} else {$8=""}; delete c; info=""; print}}' """
5245
5246                # Command - Extract only needed fields, and remove ANNOVAR fields, and compress and index final file
5247                annovar_fields_to_keep = ["INFO/ANNOVAR_DATE", "INFO/ALLELE_END"]
5248                if "ALL" not in annotation_list and "INFO" not in annotation_list:
5249                    # for ann in annotation_renamed_list:
5250                    for ann in annotation_list:
5251                        annovar_fields_to_keep.append(f"^INFO/{ann}")
5252
5253                command_annovar += f""" | {bcftools_bin_command} annotate --pair-logic exact --threads={threads} -x {",".join(annovar_fields_to_keep)} --rename-annots={tmp_rename_name} -o {tmp_annotate_vcf_name} -Oz 2>>{tmp_annotate_vcf_name_err} """
5254
5255                # Command - indexing
5256                command_annovar += f"""  && tabix {tmp_annotate_vcf_name} """
5257
5258                log.debug(f"Annotation - Annovar command: {command_annovar}")
5259                run_parallel_commands([command_annovar], 1)
5260
5261                # Error messages
5262                log.info(f"Error/Warning messages:")
5263                error_message_command_all = []
5264                error_message_command_warning = []
5265                error_message_command_err = []
5266                for err_file in err_files:
5267                    with open(err_file, "r") as f:
5268                        for line in f:
5269                            message = line.strip()
5270                            error_message_command_all.append(message)
5271                            if line.startswith("[W::") or line.startswith("WARNING"):
5272                                error_message_command_warning.append(message)
5273                            if line.startswith("[E::") or line.startswith("ERROR"):
5274                                error_message_command_err.append(
5275                                    f"{err_file}: " + message
5276                                )
5277                # log info
5278                for message in list(
5279                    set(error_message_command_err + error_message_command_warning)
5280                ):
5281                    log.info(f"   {message}")
5282                # debug info
5283                for message in list(set(error_message_command_all)):
5284                    log.debug(f"   {message}")
5285                # failed
5286                if len(error_message_command_err):
5287                    log.error("Annotation failed: Error in commands")
5288                    raise ValueError("Annotation failed: Error in commands")
5289
5290            if tmp_annotates_vcf_name_list:
5291
5292                # List of annotated files
5293                tmp_annotates_vcf_name_to_merge = " ".join(tmp_annotates_vcf_name_list)
5294
5295                # Tmp file
5296                tmp_annotate_vcf = NamedTemporaryFile(
5297                    prefix=self.get_prefix(),
5298                    dir=self.get_tmp_dir(),
5299                    suffix=".vcf.gz",
5300                    delete=False,
5301                )
5302                tmp_annotate_vcf_name = tmp_annotate_vcf.name
5303                tmp_files.append(tmp_annotate_vcf_name)
5304                tmp_annotate_vcf_name_err = tmp_annotate_vcf_name + ".err"
5305                err_files.append(tmp_annotate_vcf_name_err)
5306                tmp_files.append(tmp_annotate_vcf_name_err)
5307
5308                # Command merge
5309                merge_command = f"{bcftools_bin_command} merge --force-samples --threads={threads} {tmp_vcf_name} {tmp_annotates_vcf_name_to_merge} -o {tmp_annotate_vcf_name} -Oz 2>>{tmp_annotate_vcf_name_err} "
5310                log.info(
5311                    f"Annotation Annovar - Annotation merging "
5312                    + str(len(tmp_annotates_vcf_name_list))
5313                    + " annotated files"
5314                )
5315                log.debug(f"Annotation - merge command: {merge_command}")
5316                run_parallel_commands([merge_command], 1)
5317
5318                # Find annotation in header
5319                with bgzf.open(tmp_annotate_vcf_name, "rt") as f:
5320                    header_list = self.read_vcf_header(f)
5321                annovar_vcf_header = vcf.Reader(io.StringIO("\n".join(header_list)))
5322
5323                for ann in annovar_vcf_header.infos:
5324                    if ann not in self.get_header().infos:
5325                        vcf_reader.infos[ann] = annovar_vcf_header.infos.get(ann)
5326
5327                # Update variants
5328                log.info(f"Annotation Annovar - Updating...")
5329                self.update_from_vcf(tmp_annotate_vcf_name)
5330
5331            # Clean files
5332            # Tmp file remove command
5333            if True:
5334                tmp_files_remove_command = ""
5335                if tmp_files:
5336                    tmp_files_remove_command = " ".join(tmp_files)
5337                clean_command = f" rm -f {tmp_files_remove_command} "
5338                log.debug(f"Annotation Annovar - Annotation cleaning ")
5339                log.debug(f"Annotation - cleaning command: {clean_command}")
5340                run_parallel_commands([clean_command], 1)

It takes a VCF file, annotates it with Annovar, and then updates the database with the new annotations

Parameters

threads: number of threads to use

Returns

the value of the variable "return_value".

def annotation_parquet(self, threads: int = None) -> None: View Source

5343    def annotation_parquet(self, threads: int = None) -> None:
5344        """
5345        It takes a VCF file, and annotates it with a parquet file
5346
5347        :param threads: number of threads to use for the annotation
5348        :return: the value of the variable "result".
5349        """
5350
5351        # DEBUG
5352        log.debug("Start annotation with parquet databases")
5353
5354        # Threads
5355        if not threads:
5356            threads = self.get_threads()
5357        log.debug("Threads: " + str(threads))
5358
5359        # DEBUG
5360        delete_tmp = True
5361        if self.get_config().get("verbosity", "warning") in ["debug"]:
5362            delete_tmp = False
5363            log.debug("Delete tmp files/folders: " + str(delete_tmp))
5364
5365        # Config
5366        databases_folders = set(
5367            self.get_config()
5368            .get("folders", {})
5369            .get("databases", {})
5370            .get("annotations", ["."])
5371            + self.get_config()
5372            .get("folders", {})
5373            .get("databases", {})
5374            .get("parquet", ["."])
5375        )
5376        log.debug("Databases annotations: " + str(databases_folders))
5377
5378        # Param
5379        annotations = (
5380            self.get_param()
5381            .get("annotation", {})
5382            .get("parquet", {})
5383            .get("annotations", None)
5384        )
5385        log.debug("Annotations: " + str(annotations))
5386
5387        # Assembly
5388        assembly = self.get_param().get(
5389            "assembly", self.get_config().get("assembly", DEFAULT_ASSEMBLY)
5390        )
5391
5392        # Force Update Annotation
5393        force_update_annotation = (
5394            self.get_param()
5395            .get("annotation", {})
5396            .get("options", {})
5397            .get("annotations_update", False)
5398        )
5399        log.debug(f"force_update_annotation={force_update_annotation}")
5400        force_append_annotation = (
5401            self.get_param()
5402            .get("annotation", {})
5403            .get("options", {})
5404            .get("annotations_append", False)
5405        )
5406        log.debug(f"force_append_annotation={force_append_annotation}")
5407
5408        # Data
5409        table_variants = self.get_table_variants()
5410
5411        # Check if not empty
5412        log.debug("Check if not empty")
5413        sql_query_chromosomes_df = self.get_query_to_df(
5414            f"""SELECT count(*) as count FROM {table_variants} as table_variants LIMIT 1"""
5415        )
5416        if not sql_query_chromosomes_df["count"][0]:
5417            log.info(f"VCF empty")
5418            return
5419
5420        # VCF header
5421        vcf_reader = self.get_header()
5422        log.debug("Initial header: " + str(vcf_reader.infos))
5423
5424        # Nb Variants POS
5425        log.debug("NB Variants Start")
5426        nb_variants = self.conn.execute(
5427            f"SELECT count(*) AS count FROM variants"
5428        ).fetchdf()["count"][0]
5429        log.debug("NB Variants Stop")
5430
5431        # Existing annotations
5432        for vcf_annotation in self.get_header().infos:
5433
5434            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
5435            log.debug(
5436                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
5437            )
5438
5439        # Added columns
5440        added_columns = []
5441
5442        # drop indexes
5443        log.debug(f"Drop indexes...")
5444        self.drop_indexes()
5445
5446        if annotations:
5447
5448            if "ALL" in annotations:
5449
5450                all_param = annotations.get("ALL", {})
5451                all_param_formats = all_param.get("formats", None)
5452                all_param_releases = all_param.get("releases", None)
5453
5454                databases_infos_dict = self.scan_databases(
5455                    database_formats=all_param_formats,
5456                    database_releases=all_param_releases,
5457                )
5458                for database_infos in databases_infos_dict.keys():
5459                    if database_infos not in annotations:
5460                        annotations[database_infos] = {"INFO": None}
5461
5462            for annotation in annotations:
5463
5464                if annotation in ["ALL"]:
5465                    continue
5466
5467                # Annotation Name
5468                annotation_name = os.path.basename(annotation)
5469
5470                # Annotation fields
5471                annotation_fields = annotations[annotation]
5472                if not annotation_fields:
5473                    annotation_fields = {"INFO": None}
5474
5475                log.debug(f"Annotation '{annotation_name}'")
5476                log.debug(
5477                    f"Annotation '{annotation_name}' - fields: {annotation_fields}"
5478                )
5479
5480                # Create Database
5481                database = Database(
5482                    database=annotation,
5483                    databases_folders=databases_folders,
5484                    assembly=assembly,
5485                )
5486
5487                # Find files
5488                parquet_file = database.get_database()
5489                parquet_hdr_file = database.get_header_file()
5490                parquet_type = database.get_type()
5491
5492                # Check if files exists
5493                if not parquet_file or not parquet_hdr_file:
5494                    log.error("Annotation failed: file not found")
5495                    raise ValueError("Annotation failed: file not found")
5496                else:
5497                    # Get parquet connexion
5498                    parquet_sql_attach = database.get_sql_database_attach(
5499                        output="query"
5500                    )
5501                    if parquet_sql_attach:
5502                        self.conn.execute(parquet_sql_attach)
5503                    parquet_file_link = database.get_sql_database_link()
5504                    # Log
5505                    log.debug(
5506                        f"Annotation '{annotation_name}' - file: "
5507                        + str(parquet_file)
5508                        + " and "
5509                        + str(parquet_hdr_file)
5510                    )
5511
5512                    # Database full header columns
5513                    parquet_hdr_vcf_header_columns = database.get_header_file_columns(
5514                        parquet_hdr_file
5515                    )
5516                    # Log
5517                    log.debug(
5518                        "Annotation database header columns : "
5519                        + str(parquet_hdr_vcf_header_columns)
5520                    )
5521
5522                    # Load header as VCF object
5523                    parquet_hdr_vcf_header_infos = database.get_header().infos
5524                    # Log
5525                    log.debug(
5526                        "Annotation database header: "
5527                        + str(parquet_hdr_vcf_header_infos)
5528                    )
5529
5530                    # Get extra infos
5531                    parquet_columns = database.get_extra_columns()
5532                    # Log
5533                    log.debug("Annotation database Columns: " + str(parquet_columns))
5534
5535                    # Add extra columns if "ALL" in annotation_fields
5536                    # if "ALL" in annotation_fields:
5537                    #     allow_add_extra_column = True
5538                    if "ALL" in annotation_fields and database.get_extra_columns():
5539                        for extra_column in database.get_extra_columns():
5540                            if (
5541                                extra_column not in annotation_fields
5542                                and extra_column.replace("INFO/", "")
5543                                not in parquet_hdr_vcf_header_infos
5544                            ):
5545                                parquet_hdr_vcf_header_infos[extra_column] = (
5546                                    vcf.parser._Info(
5547                                        extra_column,
5548                                        ".",
5549                                        "String",
5550                                        f"{extra_column} description",
5551                                        "unknown",
5552                                        "unknown",
5553                                        self.code_type_map["String"],
5554                                    )
5555                                )
5556
5557                    # For all fields in database
5558                    annotation_fields_all = False
5559                    if "ALL" in annotation_fields or "INFO" in annotation_fields:
5560                        annotation_fields_all = True
5561                        annotation_fields = {
5562                            key: key for key in parquet_hdr_vcf_header_infos
5563                        }
5564
5565                        log.debug(
5566                            "Annotation database header - All annotations added: "
5567                            + str(annotation_fields)
5568                        )
5569
5570                    # Init
5571
5572                    # List of annotation fields to use
5573                    sql_query_annotation_update_info_sets = []
5574
5575                    # List of annotation to agregate
5576                    sql_query_annotation_to_agregate = []
5577
5578                    # Number of fields
5579                    nb_annotation_field = 0
5580
5581                    # Annotation fields processed
5582                    annotation_fields_processed = []
5583
5584                    # Columns mapping
5585                    map_columns = database.map_columns(
5586                        columns=annotation_fields, prefixes=["INFO/"]
5587                    )
5588
5589                    # Query dict for fields to remove (update option)
5590                    query_dict_remove = {}
5591
5592                    # Fetch Anotation fields
5593                    for annotation_field in annotation_fields:
5594
5595                        # annotation_field_column
5596                        annotation_field_column = map_columns.get(
5597                            annotation_field, "INFO"
5598                        )
5599
5600                        # field new name, if parametered
5601                        annotation_fields_new_name = annotation_fields.get(
5602                            annotation_field, annotation_field
5603                        )
5604                        if not annotation_fields_new_name:
5605                            annotation_fields_new_name = annotation_field
5606
5607                        # To annotate
5608                        # force_update_annotation = True
5609                        # force_append_annotation = True
5610                        # if annotation_field in parquet_hdr_vcf_header_infos and (force_update_annotation or (annotation_fields_new_name not in self.get_header().infos)):
5611                        if annotation_field in parquet_hdr_vcf_header_infos and (
5612                            force_update_annotation
5613                            or force_append_annotation
5614                            or (
5615                                annotation_fields_new_name
5616                                not in self.get_header().infos
5617                            )
5618                        ):
5619
5620                            # Add field to annotation to process list
5621                            annotation_fields_processed.append(
5622                                annotation_fields_new_name
5623                            )
5624
5625                            # explode infos for the field
5626                            annotation_fields_new_name_info_msg = ""
5627                            if (
5628                                force_update_annotation
5629                                and annotation_fields_new_name
5630                                in self.get_header().infos
5631                            ):
5632                                # Remove field from INFO
5633                                query = f"""
5634                                    UPDATE {table_variants} as table_variants
5635                                    SET INFO = REGEXP_REPLACE(
5636                                                concat(table_variants.INFO,''),
5637                                                ';*{annotation_fields_new_name}=[^;]*',
5638                                                ''
5639                                                )
5640                                    WHERE concat(';',table_variants.INFO) LIKE '%;{annotation_fields_new_name}=%'
5641                                """
5642                                annotation_fields_new_name_info_msg = " [update]"
5643                                query_dict_remove[
5644                                    f"remove 'INFO/{annotation_fields_new_name}'"
5645                                ] = query
5646
5647                            # Sep between fields in INFO
5648                            nb_annotation_field += 1
5649                            if nb_annotation_field > 1:
5650                                annotation_field_sep = ";"
5651                            else:
5652                                annotation_field_sep = ""
5653
5654                            log.info(
5655                                f"Annotation '{annotation_name}' - '{annotation_field}' -> '{annotation_fields_new_name}'{annotation_fields_new_name_info_msg}"
5656                            )
5657
5658                            # Add INFO field to header
5659                            parquet_hdr_vcf_header_infos_number = (
5660                                parquet_hdr_vcf_header_infos[annotation_field].num
5661                                or "."
5662                            )
5663                            parquet_hdr_vcf_header_infos_type = (
5664                                parquet_hdr_vcf_header_infos[annotation_field].type
5665                                or "String"
5666                            )
5667                            parquet_hdr_vcf_header_infos_description = (
5668                                parquet_hdr_vcf_header_infos[annotation_field].desc
5669                                or f"{annotation_field} description"
5670                            )
5671                            parquet_hdr_vcf_header_infos_source = (
5672                                parquet_hdr_vcf_header_infos[annotation_field].source
5673                                or "unknown"
5674                            )
5675                            parquet_hdr_vcf_header_infos_version = (
5676                                parquet_hdr_vcf_header_infos[annotation_field].version
5677                                or "unknown"
5678                            )
5679
5680                            vcf_reader.infos[annotation_fields_new_name] = (
5681                                vcf.parser._Info(
5682                                    annotation_fields_new_name,
5683                                    parquet_hdr_vcf_header_infos_number,
5684                                    parquet_hdr_vcf_header_infos_type,
5685                                    parquet_hdr_vcf_header_infos_description,
5686                                    parquet_hdr_vcf_header_infos_source,
5687                                    parquet_hdr_vcf_header_infos_version,
5688                                    self.code_type_map[
5689                                        parquet_hdr_vcf_header_infos_type
5690                                    ],
5691                                )
5692                            )
5693
5694                            # Append
5695                            if force_append_annotation:
5696                                query_case_when_append = f""" AND REGEXP_EXTRACT(concat(';', table_variants.INFO), ';{annotation_fields_new_name}=([^;]*)',1) IN ('','.') """
5697                            else:
5698                                query_case_when_append = ""
5699
5700                            # Annotation/Update query fields
5701                            # Found in INFO column
5702                            if (
5703                                annotation_field_column == "INFO"
5704                                and "INFO" in parquet_hdr_vcf_header_columns
5705                            ):
5706                                sql_query_annotation_update_info_sets.append(
5707                                    f"""
5708                                CASE WHEN REGEXP_EXTRACT(concat(';', table_parquet.INFO), ';{annotation_field}=([^;]*)',1) NOT IN ('','.') {query_case_when_append}
5709                                        THEN concat('{annotation_field_sep}', '{annotation_fields_new_name}=', REGEXP_EXTRACT(concat(';', table_parquet.INFO), ';{annotation_field}=([^;]*)',1))
5710                                        ELSE ''
5711                                    END
5712                                """
5713                                )
5714                            # Found in a specific column
5715                            else:
5716                                sql_query_annotation_update_info_sets.append(
5717                                    f"""
5718                                CASE WHEN table_parquet."{annotation_field_column}" NOT IN ('','.') {query_case_when_append}
5719                                        THEN concat('{annotation_field_sep}', '{annotation_fields_new_name}=', replace(table_parquet."{annotation_field_column}", ';', ','))
5720                                        ELSE ''
5721                                    END
5722                                """
5723                                )
5724                                sql_query_annotation_to_agregate.append(
5725                                    f""" string_agg(DISTINCT table_parquet_from."{annotation_field_column}", ',') AS "{annotation_field_column}" """
5726                                )
5727
5728                        # Not to annotate
5729                        else:
5730
5731                            if force_update_annotation:
5732                                annotation_message = "forced"
5733                            else:
5734                                annotation_message = "skipped"
5735
5736                            if annotation_field not in parquet_hdr_vcf_header_infos:
5737                                log.warning(
5738                                    f"Annotation '{annotation_name}' - '{annotation_field}' [{nb_annotation_field}] - not available in parquet file"
5739                                )
5740                            if annotation_fields_new_name in self.get_header().infos:
5741                                log.warning(
5742                                    f"Annotation '{annotation_name}' - '{annotation_fields_new_name}' [{nb_annotation_field}] - already exists in header ({annotation_message})"
5743                                )
5744
5745                    # Check if ALL fields have to be annotated. Thus concat all INFO field
5746                    # allow_annotation_full_info = True
5747                    allow_annotation_full_info = not force_append_annotation
5748
5749                    if parquet_type in ["regions"]:
5750                        allow_annotation_full_info = False
5751
5752                    if (
5753                        allow_annotation_full_info
5754                        and nb_annotation_field == len(annotation_fields)
5755                        and annotation_fields_all
5756                        and (
5757                            "INFO" in parquet_hdr_vcf_header_columns
5758                            and "INFO" in database.get_extra_columns()
5759                        )
5760                    ):
5761                        log.debug("Column INFO annotation enabled")
5762                        sql_query_annotation_update_info_sets = []
5763                        sql_query_annotation_update_info_sets.append(
5764                            f" table_parquet.INFO "
5765                        )
5766
5767                    if sql_query_annotation_update_info_sets:
5768
5769                        # Annotate
5770                        log.info(f"Annotation '{annotation_name}' - Annotation...")
5771
5772                        # Join query annotation update info sets for SQL
5773                        sql_query_annotation_update_info_sets_sql = ",".join(
5774                            sql_query_annotation_update_info_sets
5775                        )
5776
5777                        # Check chromosomes list (and variants infos)
5778                        sql_query_chromosomes = f"""
5779                            SELECT table_variants."#CHROM" as CHROM, count(*) AS count_variants, min(POS) AS min_variants, MAX(POS) AS max_variants
5780                            FROM {table_variants} as table_variants
5781                            GROUP BY table_variants."#CHROM"
5782                            ORDER BY table_variants."#CHROM"
5783                            """
5784                        sql_query_chromosomes_df = self.conn.execute(
5785                            sql_query_chromosomes
5786                        ).df()
5787                        sql_query_chromosomes_dict = {
5788                            entry["CHROM"]: {
5789                                "count": entry["count_variants"],
5790                                "min": entry["min_variants"],
5791                                "max": entry["max_variants"],
5792                            }
5793                            for index, entry in sql_query_chromosomes_df.iterrows()
5794                        }
5795
5796                        # Init
5797                        nb_of_query = 0
5798                        nb_of_variant_annotated = 0
5799                        query_dict = query_dict_remove
5800
5801                        # for chrom in sql_query_chromosomes_df["CHROM"]:
5802                        for chrom in sql_query_chromosomes_dict:
5803
5804                            # Number of variant by chromosome
5805                            nb_of_variant_by_chrom = sql_query_chromosomes_dict.get(
5806                                chrom, {}
5807                            ).get("count", 0)
5808
5809                            log.debug(
5810                                f"Annotation '{annotation_name}' - Chromosome '{chrom}' [{nb_of_variant_by_chrom} variants]..."
5811                            )
5812
5813                            # Annotation with regions database
5814                            if parquet_type in ["regions"]:
5815                                sql_query_annotation_from_clause = f"""
5816                                    FROM (
5817                                        SELECT 
5818                                            '{chrom}' AS \"#CHROM\",
5819                                            table_variants_from.\"POS\" AS \"POS\",
5820                                            {",".join(sql_query_annotation_to_agregate)}
5821                                        FROM {table_variants} as table_variants_from
5822                                        LEFT JOIN {parquet_file_link} as table_parquet_from ON (
5823                                            table_parquet_from."#CHROM" = '{chrom}'
5824                                            AND table_variants_from.\"POS\" <= table_parquet_from.\"END\"
5825                                            AND (table_variants_from.\"POS\" >= (table_parquet_from.\"START\"+1)
5826                                                OR table_variants_from.\"POS\" + (len(table_variants_from.\"REF\")-1) >= (table_parquet_from.\"START\"+1)
5827                                                )
5828                                        )
5829                                        WHERE table_variants_from.\"#CHROM\" in ('{chrom}')
5830                                        GROUP BY table_variants_from.\"POS\"
5831                                        )
5832                                        as table_parquet
5833                                """
5834
5835                                sql_query_annotation_where_clause = """
5836                                    table_parquet.\"#CHROM\" = table_variants.\"#CHROM\"
5837                                    AND table_parquet.\"POS\" = table_variants.\"POS\"
5838                                """
5839
5840                            # Annotation with variants database
5841                            else:
5842                                sql_query_annotation_from_clause = f"""
5843                                    FROM {parquet_file_link} as table_parquet
5844                                """
5845                                sql_query_annotation_where_clause = f"""
5846                                    table_variants."#CHROM" = '{chrom}'
5847                                    AND table_parquet.\"#CHROM\" = table_variants.\"#CHROM\" 
5848                                    AND table_parquet.\"POS\" = table_variants.\"POS\"
5849                                    AND table_parquet.\"ALT\" = table_variants.\"ALT\"
5850                                    AND table_parquet.\"REF\" = table_variants.\"REF\"
5851                                """
5852
5853                            # Create update query
5854                            sql_query_annotation_chrom_interval_pos = f"""
5855                                UPDATE {table_variants} as table_variants
5856                                    SET INFO = 
5857                                        concat(
5858                                            CASE WHEN table_variants.INFO NOT IN ('','.')
5859                                                THEN table_variants.INFO
5860                                                ELSE ''
5861                                            END
5862                                            ,
5863                                            CASE WHEN table_variants.INFO NOT IN ('','.')
5864                                                        AND (
5865                                                        concat({sql_query_annotation_update_info_sets_sql})
5866                                                        )
5867                                                        NOT IN ('','.') 
5868                                                    THEN ';'
5869                                                    ELSE ''
5870                                            END
5871                                            ,
5872                                            {sql_query_annotation_update_info_sets_sql}
5873                                            )
5874                                    {sql_query_annotation_from_clause}
5875                                    WHERE {sql_query_annotation_where_clause}
5876                                    ;
5877                                """
5878
5879                            # Add update query to dict
5880                            query_dict[
5881                                f"{chrom} [{nb_of_variant_by_chrom} variants]"
5882                            ] = sql_query_annotation_chrom_interval_pos
5883
5884                        nb_of_query = len(query_dict)
5885                        num_query = 0
5886
5887                        # SET max_expression_depth TO x
5888                        self.conn.execute("SET max_expression_depth TO 10000")
5889
5890                        for query_name in query_dict:
5891                            query = query_dict[query_name]
5892                            num_query += 1
5893                            log.info(
5894                                f"Annotation '{annotation_name}' - Annotation - Query [{num_query}/{nb_of_query}] {query_name}..."
5895                            )
5896                            result = self.conn.execute(query)
5897                            nb_of_variant_annotated_by_query = result.df()["Count"][0]
5898                            nb_of_variant_annotated += nb_of_variant_annotated_by_query
5899                            log.info(
5900                                f"Annotation '{annotation_name}' - Annotation - Query [{num_query}/{nb_of_query}] {query_name} - {nb_of_variant_annotated_by_query} variants annotated"
5901                            )
5902
5903                        log.info(
5904                            f"Annotation '{annotation_name}' - Annotation of {nb_of_variant_annotated} variants out of {nb_variants} (with {nb_of_query} queries)"
5905                        )
5906
5907                    else:
5908
5909                        log.info(
5910                            f"Annotation '{annotation_name}' - No Annotations available"
5911                        )
5912
5913                    log.debug("Final header: " + str(vcf_reader.infos))
5914
5915        # Remove added columns
5916        for added_column in added_columns:
5917            self.drop_column(column=added_column)

It takes a VCF file, and annotates it with a parquet file

Parameters

threads: number of threads to use for the annotation

Returns

the value of the variable "result".

def annotation_splice(self, threads: int = None) -> None: View Source

5919    def annotation_splice(self, threads: int = None) -> None:
5920        """
5921        This function annotate with snpEff
5922
5923        :param threads: The number of threads to use
5924        :return: the value of the variable "return_value".
5925        """
5926
5927        # DEBUG
5928        log.debug("Start annotation with splice tools")
5929
5930        # Threads
5931        if not threads:
5932            threads = self.get_threads()
5933        log.debug("Threads: " + str(threads))
5934
5935        # DEBUG
5936        delete_tmp = True
5937        if self.get_config().get("verbosity", "warning") in ["debug"]:
5938            delete_tmp = False
5939            log.debug("Delete tmp files/folders: " + str(delete_tmp))
5940
5941        # Config
5942        config = self.get_config()
5943        log.debug("Config: " + str(config))
5944        splice_config = config.get("tools", {}).get("splice", {})
5945        if not splice_config:
5946            splice_config = DEFAULT_TOOLS_BIN.get("splice", {})
5947        if not splice_config:
5948            msg_err = "No Splice tool config"
5949            log.error(msg_err)
5950            raise ValueError(msg_err)
5951        log.debug(f"splice_config={splice_config}")
5952
5953        # Config - Folders - Databases
5954        databases_folders = (
5955            config.get("folders", {}).get("databases", {}).get("splice", ["."])
5956        )
5957        log.debug("Databases annotations: " + str(databases_folders))
5958
5959        # Splice docker image
5960        splice_docker_image = splice_config.get("docker").get("image")
5961
5962        # Pull splice image if it's not already there
5963        if not check_docker_image_exists(splice_docker_image):
5964            log.warning(
5965                f"Annotation: splice docker image {splice_docker_image} not found locally, trying to pull from dockerhub"
5966            )
5967            try:
5968                command(f"docker pull {splice_config.get('docker').get('image')}")
5969            except subprocess.CalledProcessError:
5970                msg_err = f"Unable to find docker {splice_docker_image} on dockerhub"
5971                log.error(msg_err)
5972                raise ValueError(msg_err)
5973                return None
5974
5975        # Config - splice databases
5976        splice_databases = (
5977            config.get("folders", {})
5978            .get("databases", {})
5979            .get("splice", DEFAULT_SPLICE_FOLDER)
5980        )
5981        splice_databases = full_path(splice_databases)
5982
5983        # Param
5984        param = self.get_param()
5985        log.debug("Param: " + str(param))
5986
5987        # Param
5988        options = param.get("annotation", {}).get("splice", {})
5989        log.debug("Options: " + str(options))
5990
5991        # Data
5992        table_variants = self.get_table_variants()
5993
5994        # Check if not empty
5995        log.debug("Check if not empty")
5996        sql_query_chromosomes = (
5997            f"""SELECT count(*) as count FROM {table_variants} as table_variants"""
5998        )
5999        if not self.get_query_to_df(f"{sql_query_chromosomes}")["count"][0]:
6000            log.info("VCF empty")
6001            return None
6002
6003        # Export in VCF
6004        log.debug("Create initial file to annotate")
6005
6006        # Create output folder
6007        output_folder = os.path.join(self.get_tmp_dir(), f"splice-{get_random()}")
6008        if not os.path.exists(output_folder):
6009            Path(output_folder).mkdir(parents=True, exist_ok=True)
6010
6011        # Create tmp VCF file
6012        tmp_vcf = NamedTemporaryFile(
6013            prefix=self.get_prefix(),
6014            dir=output_folder,
6015            suffix=".vcf",
6016            delete=False,
6017        )
6018        tmp_vcf_name = tmp_vcf.name
6019
6020        # VCF header
6021        header = self.get_header()
6022
6023        # Existing annotations
6024        for vcf_annotation in self.get_header().infos:
6025
6026            vcf_annotation_line = self.get_header().infos.get(vcf_annotation)
6027            log.debug(
6028                f"Existing annotations in VCF: {vcf_annotation} [{vcf_annotation_line}]"
6029            )
6030
6031        # Memory limit
6032        if config.get("memory", None):
6033            memory_limit = config.get("memory", "8G").upper()
6034            # upper()
6035        else:
6036            memory_limit = "8G"
6037        log.debug(f"memory_limit: {memory_limit}")
6038
6039        # Check number of variants to annotate
6040        where_clause_regex_spliceai = r"SpliceAI_\w+"
6041        where_clause_regex_spip = r"SPiP_\w+"
6042        where_clause = f""" WHERE NOT regexp_matches("INFO", '{where_clause_regex_spliceai}') AND NOT regexp_matches("INFO", '{where_clause_regex_spip}')"""
6043        df_list_of_variants_to_annotate = self.get_query_to_df(
6044            query=f""" SELECT * FROM variants {where_clause} """
6045        )
6046        if len(df_list_of_variants_to_annotate) == 0:
6047            log.warning(
6048                f"No variants to annotate with splice. Variants probably already annotated with splice"
6049            )
6050            return None
6051        else:
6052            log.info(f"Annotation: {len(df_list_of_variants_to_annotate)} variants")
6053
6054        # Export VCF file
6055        self.export_variant_vcf(
6056            vcf_file=tmp_vcf_name,
6057            remove_info=True,
6058            add_samples=True,
6059            index=False,
6060            where_clause=where_clause,
6061        )
6062
6063        # Create docker container and launch splice analysis
6064        if splice_config:
6065
6066            # Splice mount folders
6067            mount_folders = splice_config.get("mount", {})
6068
6069            # Genome mount
6070            mount_folders[
6071                config.get("folders", {})
6072                .get("databases", {})
6073                .get("genomes", DEFAULT_GENOME_FOLDER)
6074            ] = "ro"
6075
6076            # SpliceAI mount
6077            mount_folders[
6078                config.get("folders", {})
6079                .get("databases", {})
6080                .get("spliceai", DEFAULT_SPLICEAI_FOLDER)
6081            ] = "ro"
6082
6083            # Genome mount
6084            mount_folders[
6085                config.get("folders", {})
6086                .get("databases", {})
6087                .get("spip", DEFAULT_SPIP_FOLDER)
6088            ] = "ro"
6089
6090            # Mount folders
6091            mount = []
6092
6093            # Config mount
6094            mount = [
6095                f"-v {full_path(path)}:{full_path(path)}:{mode}"
6096                for path, mode in mount_folders.items()
6097            ]
6098
6099            if any(value for value in splice_config.values() if value is None):
6100                log.warning("At least one splice config parameter is empty")
6101                return None
6102
6103            # Params in splice nf
6104            def check_values(dico: dict):
6105                """
6106                Ensure parameters for NF splice pipeline
6107                """
6108                for key, val in dico.items():
6109                    if key == "genome":
6110                        if any(
6111                            assemb in options.get("genome", {})
6112                            for assemb in ["hg19", "GRCh37", "grch37", "GRCH37"]
6113                        ):
6114                            yield f"--{key} hg19"
6115                        elif any(
6116                            assemb in options.get("genome", {})
6117                            for assemb in ["hg38", "GRCh38", "grch38", "GRCH38"]
6118                        ):
6119                            yield f"--{key} hg38"
6120                    elif (
6121                        (isinstance(val, str) and val)
6122                        or isinstance(val, int)
6123                        or isinstance(val, bool)
6124                    ):
6125                        yield f"--{key} {val}"
6126
6127            # Genome
6128            genome = options.get("genome", config.get("assembly", DEFAULT_ASSEMBLY))
6129            options["genome"] = genome
6130
6131            # NF params
6132            nf_params = []
6133
6134            # Add options
6135            if options:
6136                nf_params = list(check_values(options))
6137                log.debug(f"Splice NF params: {' '.join(nf_params)}")
6138            else:
6139                log.debug("No NF params provided")
6140
6141            # Add threads
6142            if "threads" not in options.keys():
6143                nf_params.append(f"--threads {threads}")
6144
6145            # Genome path
6146            genome_path = find_genome(
6147                config.get("folders", {})
6148                .get("databases", {})
6149                .get("genomes", DEFAULT_GENOME_FOLDER),
6150                file=f"{genome}.fa",
6151            )
6152            # Add genome path
6153            if not genome_path:
6154                raise ValueError(
6155                    f"Can't find genome assembly {genome}.fa in {config.get('folders', {}).get('databases', {}).get('genomes', DEFAULT_GENOME_FOLDER)}"
6156                )
6157            else:
6158                log.debug(f"Genome: {genome_path}")
6159                nf_params.append(f"--genome_path {genome_path}")
6160
6161            def splice_annotations(options: dict = {}, config: dict = {}) -> list:
6162                """
6163                Setting up updated databases for SPiP and SpliceAI
6164                """
6165
6166                try:
6167
6168                    # SpliceAI assembly transcriptome
6169                    spliceai_assembly = os.path.join(
6170                        config.get("folders", {})
6171                        .get("databases", {})
6172                        .get("spliceai", {}),
6173                        options.get("genome"),
6174                        "transcriptome",
6175                    )
6176                    spip_assembly = options.get("genome")
6177
6178                    spip = find(
6179                        f"transcriptome_{spip_assembly}.RData",
6180                        config.get("folders", {}).get("databases", {}).get("spip", {}),
6181                    )
6182                    spliceai = find("spliceai.refseq.txt", spliceai_assembly)
6183                    log.debug(f"SPiP annotations: {spip}")
6184                    log.debug(f"SpliceAI annotations: {spliceai}")
6185                    if spip and spliceai:
6186                        return [
6187                            f"--spip_transcriptome {spip}",
6188                            f"--spliceai_annotations {spliceai}",
6189                        ]
6190                    else:
6191                        # TODO crash and go on with basic annotations ?
6192                        # raise ValueError(
6193                        #     "Can't find splice databases in configuration EXIT"
6194                        # )
6195                        log.warning(
6196                            "Can't find splice databases in configuration, use annotations file from image"
6197                        )
6198                except TypeError:
6199                    log.warning(
6200                        "Can't find splice databases in configuration, use annotations file from image"
6201                    )
6202                    return []
6203
6204            # Add options, check if transcriptome option have already beend provided
6205            if (
6206                "spip_transcriptome" not in nf_params
6207                and "spliceai_transcriptome" not in nf_params
6208            ):
6209                splice_reference = splice_annotations(options, config)
6210                if splice_reference:
6211                    nf_params.extend(splice_reference)
6212
6213            nf_params.append(f"--output_folder {output_folder}")
6214
6215            random_uuid = f"HOWARD-SPLICE-{get_random()}"
6216            cmd = f"nextflow -log {os.path.join(output_folder, f'{random_uuid}.log')} -c /app/SpliceToolBox/src/splicetoolbox/nextflow/nextflow.docker.config run /app/SpliceToolBox/src/splicetoolbox/nextflow/main.nf -entry SPLICE --vcf {tmp_vcf_name} {' '.join(nf_params)} -profile standard,conda,singularity,report,timeline"
6217            log.debug(cmd)
6218
6219            splice_config["docker"]["command"] = cmd
6220
6221            docker_cmd = get_bin_command(
6222                tool="splice",
6223                bin_type="docker",
6224                config=config,
6225                default_folder=f"{DEFAULT_TOOLS_FOLDER}/docker",
6226                add_options=f"--name {random_uuid} {' '.join(mount)}",
6227            )
6228
6229            # Docker debug
6230            # if splice_config.get("rm_container"):
6231            #     rm_container = "--rm"
6232            # else:
6233            #     rm_container = ""
6234            # docker_cmd = f"docker run {rm_container} --entrypoint '/bin/bash' --name {random_uuid} {' '.join(mount)} {':'.join(splice_config.get('image'))} {cmd}"
6235
6236            log.debug(docker_cmd)
6237            res = subprocess.run(docker_cmd, shell=True, capture_output=True, text=True)
6238            log.debug(res.stdout)
6239            if res.stderr:
6240                log.error(res.stderr)
6241            res.check_returncode()
6242        else:
6243            log.warning(f"Splice tool configuration not found: {config}")
6244
6245        # Update variants
6246        log.info("Annotation - Updating...")
6247        # Test find output vcf
6248        log.debug(
6249            f"TMP splice output: {os.path.basename(tmp_vcf_name).replace('.vcf', '')}.spip.spliceai.sorted.vcf.gz"
6250        )
6251        output_vcf = []
6252        # Wrong folder to look in
6253        for files in os.listdir(os.path.dirname(tmp_vcf_name)):
6254            if (
6255                files
6256                == f"{os.path.basename(tmp_vcf_name).replace('.vcf', '')}.spip.spliceai.sorted.vcf.gz"
6257            ):
6258                output_vcf.append(os.path.join(os.path.dirname(tmp_vcf_name), files))
6259        # log.debug(os.listdir(options.get("output_folder")))
6260        log.debug(f"Splice annotated vcf: {output_vcf[0]}")
6261        if not output_vcf:
6262            log.debug(
6263                f"Splice output was not generated {os.path.basename(tmp_vcf_name)}*.spip.spliceai.sorted.vcf.gz"
6264            )
6265        else:
6266            # Get new header from annotated vcf
6267            log.debug(f"Initial header: {len(header.infos)} fields")
6268            # Create new header with splice infos
6269            new_vcf = Variants(input=output_vcf[0])
6270            new_vcf_header = new_vcf.get_header().infos
6271            for keys, infos in new_vcf_header.items():
6272                if keys not in header.infos.keys():
6273                    header.infos[keys] = infos
6274            log.debug(f"New header: {len(header.infos)} fields")
6275            log.debug(f"Splice tmp output: {output_vcf[0]}")
6276            self.update_from_vcf(output_vcf[0])
6277
6278        # Remove folder
6279        remove_if_exists(output_folder)

This function annotate with snpEff

Parameters

threads: The number of threads to use

Returns

the value of the variable "return_value".

def get_config_default(self, name: str) -> dict: View Source

6285    def get_config_default(self, name: str) -> dict:
6286        """
6287        The function `get_config_default` returns a dictionary containing default configurations for
6288        various calculations and prioritizations.
6289
6290        :param name: The `get_config_default` function returns a dictionary containing default
6291        configurations for different calculations and prioritizations. The `name` parameter is used to
6292        specify which specific configuration to retrieve from the dictionary
6293        :type name: str
6294        :return: The function `get_config_default` returns a dictionary containing default configuration
6295        settings for different calculations and prioritizations. The specific configuration settings are
6296        retrieved based on the input `name` parameter provided to the function. If the `name` parameter
6297        matches a key in the `config_default` dictionary, the corresponding configuration settings are
6298        returned. If there is no match, an empty dictionary is returned.
6299        """
6300
6301        config_default = {
6302            "calculations": {
6303                "variant_chr_pos_alt_ref": {
6304                    "type": "sql",
6305                    "name": "variant_chr_pos_alt_ref",
6306                    "description": "Create a variant ID with chromosome, position, alt and ref",
6307                    "available": False,
6308                    "output_column_name": "variant_chr_pos_alt_ref",
6309                    "output_column_type": "String",
6310                    "output_column_description": "variant ID with chromosome, position, alt and ref",
6311                    "operation_query": """ concat("#CHROM", '_', "POS", '_', "REF", '_', "ALT") """,
6312                    "operation_info": True,
6313                },
6314                "VARTYPE": {
6315                    "type": "sql",
6316                    "name": "VARTYPE",
6317                    "description": "Variant type (e.g. SNV, INDEL, MNV, BND...)",
6318                    "available": True,
6319                    "output_column_name": "VARTYPE",
6320                    "output_column_type": "String",
6321                    "output_column_description": "Variant type: SNV if X>Y, MOSAIC if X>Y,Z or X,Y>Z, INDEL if XY>Z or X>YZ",
6322                    "operation_query": """
6323                            CASE
6324                                WHEN "SVTYPE" NOT NULL THEN "SVTYPE"
6325                                WHEN LENGTH(REF) = 1 AND LENGTH(ALT) = 1 THEN 'SNV'
6326                                WHEN REF LIKE '%,%' OR ALT LIKE '%,%' THEN 'MOSAIC'
6327                                WHEN LENGTH(REF) == LENGTH(ALT) AND LENGTH(REF) > 1 THEN 'MNV'
6328                                WHEN LENGTH(REF) <> LENGTH(ALT) THEN 'INDEL'
6329                                ELSE 'UNDEFINED'
6330                            END
6331                            """,
6332                    "info_fields": ["SVTYPE"],
6333                    "operation_info": True,
6334                },
6335                "snpeff_hgvs": {
6336                    "type": "python",
6337                    "name": "snpeff_hgvs",
6338                    "description": "HGVS nomenclatures from snpEff annotation",
6339                    "available": True,
6340                    "function_name": "calculation_extract_snpeff_hgvs",
6341                    "function_params": ["snpeff_hgvs", "ANN"],
6342                },
6343                "snpeff_ann_explode": {
6344                    "type": "python",
6345                    "name": "snpeff_ann_explode",
6346                    "description": "Explode snpEff annotations with uniquify values",
6347                    "available": True,
6348                    "function_name": "calculation_snpeff_ann_explode",
6349                    "function_params": [False, "fields", "snpeff_", "ANN"],
6350                },
6351                "snpeff_ann_explode_uniquify": {
6352                    "type": "python",
6353                    "name": "snpeff_ann_explode_uniquify",
6354                    "description": "Explode snpEff annotations",
6355                    "available": True,
6356                    "function_name": "calculation_snpeff_ann_explode",
6357                    "function_params": [True, "fields", "snpeff_uniquify_", "ANN"],
6358                },
6359                "snpeff_ann_explode_json": {
6360                    "type": "python",
6361                    "name": "snpeff_ann_explode_json",
6362                    "description": "Explode snpEff annotations in JSON format",
6363                    "available": True,
6364                    "function_name": "calculation_snpeff_ann_explode",
6365                    "function_params": [False, "JSON", "snpeff_json", "ANN"],
6366                },
6367                "NOMEN": {
6368                    "type": "python",
6369                    "name": "NOMEN",
6370                    "description": "NOMEN information (e.g. NOMEN, CNOMEN, PNOMEN...) from HGVS nomenclature field",
6371                    "available": True,
6372                    "function_name": "calculation_extract_nomen",
6373                    "function_params": [],
6374                },
6375                "FINDBYPIPELINE": {
6376                    "type": "python",
6377                    "name": "FINDBYPIPELINE",
6378                    "description": "Number of pipeline that identify the variant (for multi pipeline VCF)",
6379                    "available": True,
6380                    "function_name": "calculation_find_by_pipeline",
6381                    "function_params": ["findbypipeline"],
6382                },
6383                "FINDBYSAMPLE": {
6384                    "type": "python",
6385                    "name": "FINDBYSAMPLE",
6386                    "description": "Number of sample that have a genotype for the variant (for multi sample VCF)",
6387                    "available": True,
6388                    "function_name": "calculation_find_by_pipeline",
6389                    "function_params": ["findbysample"],
6390                },
6391                "GENOTYPECONCORDANCE": {
6392                    "type": "python",
6393                    "name": "GENOTYPECONCORDANCE",
6394                    "description": "Concordance of genotype for multi caller VCF",
6395                    "available": True,
6396                    "function_name": "calculation_genotype_concordance",
6397                    "function_params": [],
6398                },
6399                "BARCODE": {
6400                    "type": "python",
6401                    "name": "BARCODE",
6402                    "description": "BARCODE as VaRank tool",
6403                    "available": True,
6404                    "function_name": "calculation_barcode",
6405                    "function_params": [],
6406                },
6407                "BARCODEFAMILY": {
6408                    "type": "python",
6409                    "name": "BARCODEFAMILY",
6410                    "description": "BARCODEFAMILY as VaRank tool",
6411                    "available": True,
6412                    "function_name": "calculation_barcode_family",
6413                    "function_params": ["BCF"],
6414                },
6415                "TRIO": {
6416                    "type": "python",
6417                    "name": "TRIO",
6418                    "description": "Inheritance for a trio family",
6419                    "available": True,
6420                    "function_name": "calculation_trio",
6421                    "function_params": [],
6422                },
6423                "VAF": {
6424                    "type": "python",
6425                    "name": "VAF",
6426                    "description": "Variant Allele Frequency (VAF) harmonization",
6427                    "available": True,
6428                    "function_name": "calculation_vaf_normalization",
6429                    "function_params": [],
6430                },
6431                "VAF_stats": {
6432                    "type": "python",
6433                    "name": "VAF_stats",
6434                    "description": "Variant Allele Frequency (VAF) statistics",
6435                    "available": True,
6436                    "function_name": "calculation_genotype_stats",
6437                    "function_params": ["VAF"],
6438                },
6439                "DP_stats": {
6440                    "type": "python",
6441                    "name": "DP_stats",
6442                    "description": "Depth (DP) statistics",
6443                    "available": True,
6444                    "function_name": "calculation_genotype_stats",
6445                    "function_params": ["DP"],
6446                },
6447                "variant_id": {
6448                    "type": "python",
6449                    "name": "variant_id",
6450                    "description": "Variant ID generated from variant position and type",
6451                    "available": True,
6452                    "function_name": "calculation_variant_id",
6453                    "function_params": [],
6454                },
6455                "transcripts_json": {
6456                    "type": "python",
6457                    "name": "transcripts_json",
6458                    "description": "Add transcripts annotations in JSON format (field 'transcripts_json')",
6459                    "available": True,
6460                    "function_name": "calculation_transcripts_annotation",
6461                    "function_params": ["transcripts_json", None],
6462                },
6463                "transcripts_ann": {
6464                    "type": "python",
6465                    "name": "transcripts_ann",
6466                    "description": "Add transcripts annotations in structured format (field 'transcripts_ann')",
6467                    "available": True,
6468                    "function_name": "calculation_transcripts_annotation",
6469                    "function_params": [None, "transcripts_ann"],
6470                },
6471                "transcripts_annotations": {
6472                    "type": "python",
6473                    "name": "transcripts_annotations",
6474                    "description": "Add transcripts annotations in JSON and/or structured format (see param JSON file)",
6475                    "available": True,
6476                    "function_name": "calculation_transcripts_annotation",
6477                    "function_params": [None, None],
6478                },
6479                "transcripts_prioritization": {
6480                    "type": "python",
6481                    "name": "transcripts_prioritization",
6482                    "description": "Prioritize transcripts with a prioritization profile (using param.json)",
6483                    "available": True,
6484                    "function_name": "calculation_transcripts_prioritization",
6485                    "function_params": [],
6486                },
6487            },
6488            "prioritizations": {
6489                "default": {
6490                    "filter": [
6491                        {
6492                            "type": "notequals",
6493                            "value": "!PASS|\\.",
6494                            "score": 0,
6495                            "flag": "FILTERED",
6496                            "comment": ["Bad variant quality"],
6497                        },
6498                        {
6499                            "type": "equals",
6500                            "value": "REJECT",
6501                            "score": -20,
6502                            "flag": "PASS",
6503                            "comment": ["Bad variant quality"],
6504                        },
6505                    ],
6506                    "DP": [
6507                        {
6508                            "type": "gte",
6509                            "value": "50",
6510                            "score": 5,
6511                            "flag": "PASS",
6512                            "comment": ["DP higher than 50"],
6513                        }
6514                    ],
6515                    "ANN": [
6516                        {
6517                            "type": "contains",
6518                            "value": "HIGH",
6519                            "score": 5,
6520                            "flag": "PASS",
6521                            "comment": [
6522                                "The variant is assumed to have high (disruptive) impact in the protein, probably causing protein truncation, loss of function or triggering nonsense mediated decay"
6523                            ],
6524                        },
6525                        {
6526                            "type": "contains",
6527                            "value": "MODERATE",
6528                            "score": 3,
6529                            "flag": "PASS",
6530                            "comment": [
6531                                "A non-disruptive variant that might change protein effectiveness"
6532                            ],
6533                        },
6534                        {
6535                            "type": "contains",
6536                            "value": "LOW",
6537                            "score": 0,
6538                            "flag": "FILTERED",
6539                            "comment": [
6540                                "Assumed to be mostly harmless or unlikely to change protein behavior"
6541                            ],
6542                        },
6543                        {
6544                            "type": "contains",
6545                            "value": "MODIFIER",
6546                            "score": 0,
6547                            "flag": "FILTERED",
6548                            "comment": [
6549                                "Usually non-coding variants or variants affecting non-coding genes, where predictions are difficult or there is no evidence of impact"
6550                            ],
6551                        },
6552                    ],
6553                }
6554            },
6555        }
6556
6557        return config_default.get(name, None)

The function get_config_default returns a dictionary containing default configurations for various calculations and prioritizations.

Parameters

name: The get_config_default function returns a dictionary containing default configurations for different calculations and prioritizations. The name parameter is used to specify which specific configuration to retrieve from the dictionary

Returns

The function get_config_default returns a dictionary containing default configuration settings for different calculations and prioritizations. The specific configuration settings are retrieved based on the input name parameter provided to the function. If the name parameter matches a key in the config_default dictionary, the corresponding configuration settings are returned. If there is no match, an empty dictionary is returned.

def get_config_json(self, name: str, config_dict: dict = {}, config_file: str = None) -> dict: View Source

6559    def get_config_json(
6560        self, name: str, config_dict: dict = {}, config_file: str = None
6561    ) -> dict:
6562        """
6563        The function `get_config_json` retrieves a configuration JSON object with prioritizations from
6564        default values, a dictionary, and a file.
6565
6566        :param name: The `name` parameter in the `get_config_json` function is a string that represents
6567        the name of the configuration. It is used to identify and retrieve the configuration settings
6568        for a specific component or module
6569        :type name: str
6570        :param config_dict: The `config_dict` parameter in the `get_config_json` function is a
6571        dictionary that allows you to provide additional configuration settings or overrides. When you
6572        call the `get_config_json` function, you can pass a dictionary containing key-value pairs where
6573        the key is the configuration setting you want to override or
6574        :type config_dict: dict
6575        :param config_file: The `config_file` parameter in the `get_config_json` function is used to
6576        specify the path to a configuration file that contains additional settings. If provided, the
6577        function will read the contents of this file and update the configuration dictionary with the
6578        values found in the file, overriding any existing values with the
6579        :type config_file: str
6580        :return: The function `get_config_json` returns a dictionary containing the configuration
6581        settings.
6582        """
6583
6584        # Create with default prioritizations
6585        config_default = self.get_config_default(name=name)
6586        configuration = config_default
6587        # log.debug(f"configuration={configuration}")
6588
6589        # Replace prioritizations from dict
6590        for config in config_dict:
6591            configuration[config] = config_dict[config]
6592
6593        # Replace prioritizations from file
6594        config_file = full_path(config_file)
6595        if config_file:
6596            if os.path.exists(config_file):
6597                with open(config_file) as config_file_content:
6598                    config_file_dict = json.load(config_file_content)
6599                for config in config_file_dict:
6600                    configuration[config] = config_file_dict[config]
6601            else:
6602                msg_error = f"Config '{name}' file '{config_file}' does NOT exist"
6603                log.error(msg_error)
6604                raise ValueError(msg_error)
6605
6606        return configuration

The function get_config_json retrieves a configuration JSON object with prioritizations from default values, a dictionary, and a file.

Parameters

name: The name parameter in the get_config_json function is a string that represents the name of the configuration. It is used to identify and retrieve the configuration settings for a specific component or module
config_dict: The config_dict parameter in the get_config_json function is a dictionary that allows you to provide additional configuration settings or overrides. When you call the get_config_json function, you can pass a dictionary containing key-value pairs where the key is the configuration setting you want to override or
config_file: The config_file parameter in the get_config_json function is used to specify the path to a configuration file that contains additional settings. If provided, the function will read the contents of this file and update the configuration dictionary with the values found in the file, overriding any existing values with the

Returns

The function get_config_json returns a dictionary containing the configuration settings.

def prioritization( self, table: str = None, pz_prefix: str = None, pz_param: dict = None) -> bool: View Source

6608    def prioritization(
6609        self, table: str = None, pz_prefix: str = None, pz_param: dict = None
6610    ) -> bool:
6611        """
6612        The `prioritization` function in Python processes VCF files, adds new INFO fields, and
6613        prioritizes variants based on configured profiles and criteria.
6614
6615        :param table: The `table` parameter in the `prioritization` function is used to specify the name
6616        of the table (presumably a VCF file) on which the prioritization operation will be performed. If
6617        a table name is provided, the method will prioritize the variants in that specific table
6618        :type table: str
6619        :param pz_prefix: The `pz_prefix` parameter is used to specify a prefix that will be added to
6620        certain INFO fields in a VCF file during the prioritization process. If this parameter is not
6621        provided, the code will use a default prefix value of "PZ"
6622        :type pz_prefix: str
6623        :param pz_param: The `pz_param` parameter in the `prioritization` method is used to pass
6624        additional parameters specific to the prioritization process. These parameters can include
6625        settings related to prioritization profiles, fields, scoring modes, flags, comments, and other
6626        configurations needed for the prioritization of variants in a V
6627        :type pz_param: dict
6628        :return: A boolean value (True) is being returned from the `prioritization` function.
6629        """
6630
6631        # Config
6632        config = self.get_config()
6633
6634        # Param
6635        param = self.get_param()
6636
6637        # Prioritization param
6638        if pz_param is not None:
6639            prioritization_param = pz_param
6640        else:
6641            prioritization_param = param.get("prioritization", {})
6642
6643        # Configuration profiles
6644        prioritization_config_file = prioritization_param.get(
6645            "prioritization_config", None
6646        )
6647        prioritization_config_file = full_path(prioritization_config_file)
6648        prioritizations_config = self.get_config_json(
6649            name="prioritizations", config_file=prioritization_config_file
6650        )
6651
6652        # Prioritization prefix
6653        pz_prefix_default = "PZ"
6654        if pz_prefix is None:
6655            pz_prefix = prioritization_param.get("pzprefix", pz_prefix_default)
6656
6657        # Prioritization options
6658        profiles = prioritization_param.get("profiles", [])
6659        if isinstance(profiles, str):
6660            profiles = profiles.split(",")
6661        pzfields = prioritization_param.get(
6662            "pzfields", [f"{pz_prefix}Flag", f"{pz_prefix}Score"]
6663        )
6664        if isinstance(pzfields, str):
6665            pzfields = pzfields.split(",")
6666        default_profile = prioritization_param.get("default_profile", None)
6667        pzfields_sep = prioritization_param.get("pzfields_sep", "_")
6668        prioritization_score_mode = prioritization_param.get(
6669            "prioritization_score_mode", "HOWARD"
6670        )
6671
6672        # Quick Prioritizations
6673        prioritizations = param.get("prioritizations", None)
6674        if prioritizations:
6675            log.info("Quick Prioritization:")
6676            for profile in prioritizations.split(","):
6677                if profile not in profiles:
6678                    profiles.append(profile)
6679                    log.info(f"   {profile}")
6680
6681        # If profile "ALL" provided, all profiles in the config profiles
6682        if "ALL" in profiles:
6683            profiles = list(prioritizations_config.keys())
6684
6685        for profile in profiles:
6686            if prioritizations_config.get(profile, None):
6687                log.debug(f"Profile '{profile}' configured")
6688            else:
6689                msg_error = f"Profile '{profile}' NOT configured"
6690                log.error(msg_error)
6691                raise ValueError(msg_error)
6692
6693        if profiles:
6694            log.info(f"Prioritization... ")
6695        else:
6696            log.debug(f"No profile defined")
6697            return False
6698
6699        if not default_profile and len(profiles):
6700            default_profile = profiles[0]
6701
6702        log.debug("Profiles availables: " + str(list(prioritizations_config.keys())))
6703        log.debug("Profiles to check: " + str(list(profiles)))
6704
6705        # Variables
6706        if table is not None:
6707            table_variants = table
6708        else:
6709            table_variants = self.get_table_variants(clause="update")
6710        log.debug(f"Table to prioritize: {table_variants}")
6711
6712        # Added columns
6713        added_columns = []
6714
6715        # Create list of PZfields
6716        # List of PZFields
6717        list_of_pzfields_original = pzfields + [
6718            pzfield + pzfields_sep + profile
6719            for pzfield in pzfields
6720            for profile in profiles
6721        ]
6722        list_of_pzfields = []
6723        log.debug(f"{list_of_pzfields_original}")
6724
6725        # Remove existing PZfields to use if exists
6726        for pzfield in list_of_pzfields_original:
6727            if self.get_header().infos.get(pzfield, None) is None:
6728                list_of_pzfields.append(pzfield)
6729                log.debug(f"VCF Input - Header - PZfield '{pzfield}' not in VCF")
6730            else:
6731                log.debug(f"VCF Input - Header - PZfield '{pzfield}' already in VCF")
6732
6733        if list_of_pzfields:
6734
6735            # Explode Infos prefix
6736            explode_infos_prefix = self.get_explode_infos_prefix()
6737
6738            # PZfields tags description
6739            PZfields_INFOS = {
6740                f"{pz_prefix}Tags": {
6741                    "ID": f"{pz_prefix}Tags",
6742                    "Number": ".",
6743                    "Type": "String",
6744                    "Description": "Variant tags based on annotation criteria",
6745                },
6746                f"{pz_prefix}Score": {
6747                    "ID": f"{pz_prefix}Score",
6748                    "Number": 1,
6749                    "Type": "Integer",
6750                    "Description": "Variant score based on annotation criteria",
6751                },
6752                f"{pz_prefix}Flag": {
6753                    "ID": f"{pz_prefix}Flag",
6754                    "Number": 1,
6755                    "Type": "String",
6756                    "Description": "Variant flag based on annotation criteria",
6757                },
6758                f"{pz_prefix}Comment": {
6759                    "ID": f"{pz_prefix}Comment",
6760                    "Number": ".",
6761                    "Type": "String",
6762                    "Description": "Variant comment based on annotation criteria",
6763                },
6764                f"{pz_prefix}Infos": {
6765                    "ID": f"{pz_prefix}Infos",
6766                    "Number": ".",
6767                    "Type": "String",
6768                    "Description": "Variant infos based on annotation criteria",
6769                },
6770            }
6771
6772            # Create INFO fields if not exist
6773            for field in PZfields_INFOS:
6774                field_ID = PZfields_INFOS[field]["ID"]
6775                field_description = PZfields_INFOS[field]["Description"]
6776                if field_ID not in self.get_header().infos and field_ID in pzfields:
6777                    field_description = (
6778                        PZfields_INFOS[field]["Description"]
6779                        + f", profile {default_profile}"
6780                    )
6781                    self.get_header().infos[field_ID] = vcf.parser._Info(
6782                        field_ID,
6783                        PZfields_INFOS[field]["Number"],
6784                        PZfields_INFOS[field]["Type"],
6785                        field_description,
6786                        "unknown",
6787                        "unknown",
6788                        code_type_map[PZfields_INFOS[field]["Type"]],
6789                    )
6790
6791            # Create INFO fields if not exist for each profile
6792            for profile in prioritizations_config:
6793                if profile in profiles or profiles == []:
6794                    for field in PZfields_INFOS:
6795                        field_ID = PZfields_INFOS[field]["ID"] + pzfields_sep + profile
6796                        field_description = (
6797                            PZfields_INFOS[field]["Description"]
6798                            + f", profile {profile}"
6799                        )
6800                        if (
6801                            field_ID not in self.get_header().infos
6802                            and field in pzfields
6803                        ):
6804                            self.get_header().infos[field_ID] = vcf.parser._Info(
6805                                field_ID,
6806                                PZfields_INFOS[field]["Number"],
6807                                PZfields_INFOS[field]["Type"],
6808                                field_description,
6809                                "unknown",
6810                                "unknown",
6811                                code_type_map[PZfields_INFOS[field]["Type"]],
6812                            )
6813
6814            # Header
6815            for pzfield in list_of_pzfields:
6816                if re.match(f"{pz_prefix}Score.*", pzfield):
6817                    added_column = self.add_column(
6818                        table_name=table_variants,
6819                        column_name=pzfield,
6820                        column_type="INTEGER",
6821                        default_value="0",
6822                    )
6823                elif re.match(f"{pz_prefix}Flag.*", pzfield):
6824                    added_column = self.add_column(
6825                        table_name=table_variants,
6826                        column_name=pzfield,
6827                        column_type="BOOLEAN",
6828                        default_value="1",
6829                    )
6830                else:
6831                    added_column = self.add_column(
6832                        table_name=table_variants,
6833                        column_name=pzfield,
6834                        column_type="STRING",
6835                        default_value="''",
6836                    )
6837                added_columns.append(added_column)
6838
6839            # Profiles
6840            if profiles:
6841
6842                # foreach profile in configuration file
6843                for profile in prioritizations_config:
6844
6845                    # If profile is asked in param, or ALL are asked (empty profile [])
6846                    if profile in profiles or profiles == []:
6847                        log.info(f"Profile '{profile}'")
6848
6849                        sql_set_info_option = ""
6850
6851                        sql_set_info = []
6852
6853                        # PZ fields set
6854
6855                        # PZScore
6856                        if (
6857                            f"{pz_prefix}Score{pzfields_sep}{profile}"
6858                            in list_of_pzfields
6859                        ):
6860                            sql_set_info.append(
6861                                f"""
6862                                    concat(
6863                                        '{pz_prefix}Score{pzfields_sep}{profile}=',
6864                                        {pz_prefix}Score{pzfields_sep}{profile}
6865                                    ) 
6866                                """
6867                            )
6868                            if (
6869                                profile == default_profile
6870                                and f"{pz_prefix}Score" in list_of_pzfields
6871                            ):
6872                                sql_set_info.append(
6873                                    f"""
6874                                        concat(
6875                                            '{pz_prefix}Score=',
6876                                            {pz_prefix}Score{pzfields_sep}{profile}
6877                                        )
6878                                    """
6879                                )
6880
6881                        # PZFlag
6882                        if (
6883                            f"{pz_prefix}Flag{pzfields_sep}{profile}"
6884                            in list_of_pzfields
6885                        ):
6886                            sql_set_info.append(
6887                                f"""
6888                                    concat(
6889                                        '{pz_prefix}Flag{pzfields_sep}{profile}=',
6890                                        CASE 
6891                                            WHEN {pz_prefix}Flag{pzfields_sep}{profile}==1
6892                                            THEN 'PASS'
6893                                            WHEN {pz_prefix}Flag{pzfields_sep}{profile}==0
6894                                            THEN 'FILTERED'
6895                                        END
6896                                    ) 
6897                                """
6898                            )
6899                            if (
6900                                profile == default_profile
6901                                and f"{pz_prefix}Flag" in list_of_pzfields
6902                            ):
6903                                sql_set_info.append(
6904                                    f"""
6905                                        concat(
6906                                            '{pz_prefix}Flag=',
6907                                            CASE 
6908                                                WHEN {pz_prefix}Flag{pzfields_sep}{profile}==1
6909                                                THEN 'PASS'
6910                                                WHEN {pz_prefix}Flag{pzfields_sep}{profile}==0
6911                                                THEN 'FILTERED'
6912                                            END
6913                                        )
6914                                    """
6915                                )
6916
6917                        # PZComment
6918                        if (
6919                            f"{pz_prefix}Comment{pzfields_sep}{profile}"
6920                            in list_of_pzfields
6921                        ):
6922                            sql_set_info.append(
6923                                f"""
6924                                    CASE
6925                                        WHEN {pz_prefix}Comment{pzfields_sep}{profile} NOT IN ('')
6926                                        THEN concat('{pz_prefix}Comment{pzfields_sep}{profile}=', {pz_prefix}Comment{pzfields_sep}{profile})
6927                                        ELSE ''
6928                                    END
6929                                """
6930                            )
6931                            if (
6932                                profile == default_profile
6933                                and f"{pz_prefix}Comment" in list_of_pzfields
6934                            ):
6935                                sql_set_info.append(
6936                                    f"""
6937                                        CASE
6938                                            WHEN {pz_prefix}Comment{pzfields_sep}{profile} NOT IN ('')
6939                                            THEN concat('{pz_prefix}Comment=', {pz_prefix}Comment{pzfields_sep}{profile})
6940                                            ELSE ''
6941                                        END
6942                                    """
6943                                )
6944
6945                        # PZInfos
6946                        if (
6947                            f"{pz_prefix}Infos{pzfields_sep}{profile}"
6948                            in list_of_pzfields
6949                        ):
6950                            sql_set_info.append(
6951                                f"""
6952                                    CASE
6953                                        WHEN {pz_prefix}Infos{pzfields_sep}{profile} NOT IN ('')
6954                                        THEN concat('{pz_prefix}Infos{pzfields_sep}{profile}=', {pz_prefix}Infos{pzfields_sep}{profile})
6955                                        ELSE ''
6956                                    END
6957                                """
6958                            )
6959                            if (
6960                                profile == default_profile
6961                                and f"{pz_prefix}Infos" in list_of_pzfields
6962                            ):
6963                                sql_set_info.append(
6964                                    f"""
6965                                        CASE
6966                                            WHEN {pz_prefix}Infos{pzfields_sep}{profile} NOT IN ('')
6967                                            THEN concat('{pz_prefix}Infos=', {pz_prefix}Infos{pzfields_sep}{profile})
6968                                            ELSE ''
6969                                        END
6970                                    """
6971                                )
6972
6973                        # Merge PZfields
6974                        sql_set_info_option = ""
6975                        sql_set_sep = ""
6976                        for sql_set in sql_set_info:
6977                            if sql_set_sep:
6978                                sql_set_info_option += f"""
6979                                    , concat('{sql_set_sep}', {sql_set})
6980                                """
6981                            else:
6982                                sql_set_info_option += f"""
6983                                    , {sql_set}
6984                                """
6985                            sql_set_sep = ";"
6986
6987                        sql_queries = []
6988                        for annotation in prioritizations_config[profile]:
6989
6990                            # Explode specific annotation
6991                            log.debug(f"Explode annotation '{annotation}'")
6992                            added_columns += self.explode_infos(
6993                                prefix=explode_infos_prefix,
6994                                fields=[annotation],
6995                                table=table_variants,
6996                            )
6997                            extra_infos = self.get_extra_infos(table=table_variants)
6998
6999                            # Check if annotation field is present
7000                            if not f"{explode_infos_prefix}{annotation}" in extra_infos:
7001                                log.debug(f"Annotation '{annotation}' not in data")
7002                                continue
7003                            else:
7004                                log.debug(f"Annotation '{annotation}' in data")
7005
7006                            # For each criterions
7007                            for criterion in prioritizations_config[profile][
7008                                annotation
7009                            ]:
7010                                criterion_type = criterion["type"]
7011                                criterion_value = criterion["value"]
7012                                criterion_score = criterion.get("score", 0)
7013                                criterion_flag = criterion.get("flag", "PASS")
7014                                criterion_flag_bool = criterion_flag == "PASS"
7015                                criterion_comment = (
7016                                    ", ".join(criterion.get("comment", []))
7017                                    .replace("'", "''")
7018                                    .replace(";", ",")
7019                                    .replace("\t", " ")
7020                                )
7021                                criterion_infos = (
7022                                    str(criterion)
7023                                    .replace("'", "''")
7024                                    .replace(";", ",")
7025                                    .replace("\t", " ")
7026                                )
7027
7028                                sql_set = []
7029                                sql_set_info = []
7030
7031                                # PZ fields set
7032                                if (
7033                                    f"{pz_prefix}Score{pzfields_sep}{profile}"
7034                                    in list_of_pzfields
7035                                ):
7036                                    if prioritization_score_mode == "HOWARD":
7037                                        sql_set.append(
7038                                            f"{pz_prefix}Score{pzfields_sep}{profile} = {pz_prefix}Score{pzfields_sep}{profile} + {criterion_score}"
7039                                        )
7040                                    elif prioritization_score_mode == "VaRank":
7041                                        sql_set.append(
7042                                            f"{pz_prefix}Score{pzfields_sep}{profile} = CASE WHEN {criterion_score}>{pz_prefix}Score{pzfields_sep}{profile} THEN {criterion_score} END"
7043                                        )
7044                                    else:
7045                                        sql_set.append(
7046                                            f"{pz_prefix}Score{pzfields_sep}{profile} = {pz_prefix}Score{pzfields_sep}{profile} + {criterion_score}"
7047                                        )
7048                                if (
7049                                    f"{pz_prefix}Flag{pzfields_sep}{profile}"
7050                                    in list_of_pzfields
7051                                ):
7052                                    sql_set.append(
7053                                        f"{pz_prefix}Flag{pzfields_sep}{profile} = {pz_prefix}Flag{pzfields_sep}{profile} AND {criterion_flag_bool}"
7054                                    )
7055                                if (
7056                                    f"{pz_prefix}Comment{pzfields_sep}{profile}"
7057                                    in list_of_pzfields
7058                                ):
7059                                    sql_set.append(
7060                                        f"""
7061                                            {pz_prefix}Comment{pzfields_sep}{profile} = 
7062                                                concat(
7063                                                    {pz_prefix}Comment{pzfields_sep}{profile},
7064                                                    CASE 
7065                                                        WHEN {pz_prefix}Comment{pzfields_sep}{profile}!=''
7066                                                        THEN ', '
7067                                                        ELSE ''
7068                                                    END,
7069                                                    '{criterion_comment}'
7070                                                )
7071                                        """
7072                                    )
7073                                if (
7074                                    f"{pz_prefix}Infos{pzfields_sep}{profile}"
7075                                    in list_of_pzfields
7076                                ):
7077                                    sql_set.append(
7078                                        f"""
7079                                            {pz_prefix}Infos{pzfields_sep}{profile} = 
7080                                                concat(
7081                                                    {pz_prefix}Infos{pzfields_sep}{profile},
7082                                                    '{criterion_infos}'
7083                                                )
7084                                        """
7085                                    )
7086                                sql_set_option = ",".join(sql_set)
7087
7088                                # Criterion and comparison
7089                                if sql_set_option:
7090                                    try:
7091                                        float(criterion_value)
7092                                        sql_update = f"""
7093                                            UPDATE {table_variants}
7094                                            SET {sql_set_option}
7095                                            WHERE CAST("{explode_infos_prefix}{annotation}" AS VARCHAR) NOT IN ('','.')
7096                                            AND CAST("{explode_infos_prefix}{annotation}" AS FLOAT){comparison_map[criterion_type]}{criterion_value}
7097                                            """
7098                                    except:
7099                                        contains_option = ""
7100                                        if criterion_type == "contains":
7101                                            contains_option = ".*"
7102                                        sql_update = f"""
7103                                            UPDATE {table_variants}
7104                                            SET {sql_set_option}
7105                                            WHERE "{explode_infos_prefix}{annotation}" SIMILAR TO '{contains_option}{criterion_value}{contains_option}'
7106                                            """
7107                                    sql_queries.append(sql_update)
7108                                else:
7109                                    log.warning(
7110                                        f"NO SQL SET option for '{annotation}' - '{criterion}'"
7111                                    )
7112
7113                        # PZTags
7114                        if (
7115                            f"{pz_prefix}Tags{pzfields_sep}{profile}"
7116                            in list_of_pzfields
7117                        ):
7118
7119                            # Create PZFalgs value
7120                            pztags_value = ""
7121                            pztags_sep_default = "|"
7122                            pztags_sep = ""
7123                            for pzfield in pzfields:
7124                                if pzfield not in [f"{pz_prefix}Tags"]:
7125                                    if (
7126                                        f"{pzfield}{pzfields_sep}{profile}"
7127                                        in list_of_pzfields
7128                                    ):
7129                                        if pzfield in [f"{pz_prefix}Flag"]:
7130                                            pztags_value += f"""{pztags_sep}{pzfield}#', 
7131                                                CASE WHEN {pz_prefix}Flag{pzfields_sep}{profile}
7132                                                    THEN 'PASS'
7133                                                    ELSE 'FILTERED'
7134                                                END, '"""
7135                                        else:
7136                                            pztags_value += f"{pztags_sep}{pzfield}#', {pzfield}{pzfields_sep}{profile}, '"
7137                                        pztags_sep = pztags_sep_default
7138
7139                            # Add Query update for PZFlags
7140                            sql_update_pztags = f"""
7141                                UPDATE {table_variants}
7142                                SET INFO = concat(
7143                                        INFO,
7144                                        CASE WHEN INFO NOT in ('','.')
7145                                                THEN ';'
7146                                                ELSE ''
7147                                        END,
7148                                        '{pz_prefix}Tags{pzfields_sep}{profile}={pztags_value}'
7149                                    )
7150                                """
7151                            sql_queries.append(sql_update_pztags)
7152
7153                            # Add Query update for PZFlags for default
7154                            if profile == default_profile:
7155                                sql_update_pztags_default = f"""
7156                                UPDATE {table_variants}
7157                                SET INFO = concat(
7158                                        INFO,
7159                                        ';',
7160                                        '{pz_prefix}Tags={pztags_value}'
7161                                    )
7162                                """
7163                                sql_queries.append(sql_update_pztags_default)
7164
7165                        log.info(f"""Profile '{profile}' - Prioritization... """)
7166
7167                        if sql_queries:
7168
7169                            for sql_query in sql_queries:
7170                                log.debug(
7171                                    f"""Profile '{profile}' - Prioritization query: {sql_query}... """
7172                                )
7173                                self.conn.execute(sql_query)
7174
7175                        log.info(f"""Profile '{profile}' - Update... """)
7176                        sql_query_update = f"""
7177                            UPDATE {table_variants}
7178                            SET INFO =  
7179                                concat(
7180                                    CASE
7181                                        WHEN INFO NOT IN ('','.')
7182                                        THEN concat(INFO, ';')
7183                                        ELSE ''
7184                                    END
7185                                    {sql_set_info_option}
7186                                )
7187                        """
7188                        self.conn.execute(sql_query_update)
7189
7190        else:
7191
7192            log.warning(f"No profiles in parameters")
7193
7194        # Remove added columns
7195        for added_column in added_columns:
7196            self.drop_column(column=added_column)
7197
7198        # Explode INFOS fields into table fields
7199        if self.get_explode_infos():
7200            self.explode_infos(
7201                prefix=self.get_explode_infos_prefix(),
7202                fields=self.get_explode_infos_fields(),
7203                force=True,
7204            )
7205
7206        return True

The prioritization function in Python processes VCF files, adds new INFO fields, and prioritizes variants based on configured profiles and criteria.

Parameters

table: The table parameter in the prioritization function is used to specify the name of the table (presumably a VCF file) on which the prioritization operation will be performed. If a table name is provided, the method will prioritize the variants in that specific table
pz_prefix: The pz_prefix parameter is used to specify a prefix that will be added to certain INFO fields in a VCF file during the prioritization process. If this parameter is not provided, the code will use a default prefix value of "PZ"
pz_param: The pz_param parameter in the prioritization method is used to pass additional parameters specific to the prioritization process. These parameters can include settings related to prioritization profiles, fields, scoring modes, flags, comments, and other configurations needed for the prioritization of variants in a V

Returns

A boolean value (True) is being returned from the prioritization function.

def annotation_hgvs(self, threads: int = None) -> None: View Source

7212    def annotation_hgvs(self, threads: int = None) -> None:
7213        """
7214        The `annotation_hgvs` function performs HGVS annotation on a set of variants using genomic
7215        coordinates and alleles.
7216
7217        :param threads: The `threads` parameter is an optional integer that specifies the number of
7218        threads to use for parallel processing. If no value is provided, it will default to the number
7219        of threads obtained from the `get_threads()` method
7220        :type threads: int
7221        """
7222
7223        # Function for each partition of the Dask Dataframe
7224        def partition_function(partition):
7225            """
7226            The function `partition_function` applies the `annotation_hgvs_partition` function to
7227            each row of a DataFrame called `partition`.
7228
7229            :param partition: The parameter "partition" is a pandas DataFrame that contains the data
7230            to be processed
7231            :return: the result of applying the "annotation_hgvs_partition" function to each row of
7232            the "partition" dataframe along the axis 1.
7233            """
7234            return partition.apply(annotation_hgvs_partition, axis=1)
7235
7236        def annotation_hgvs_partition(row) -> str:
7237            """
7238            The function `annotation_hgvs_partition` takes in a row of data and returns a string
7239            containing a list of HGVS names associated with the given genomic coordinates and alleles.
7240
7241            :param row: A dictionary-like object that contains the values for the following keys:
7242            :return: a string that contains the HGVS names associated with the given row of data.
7243            """
7244
7245            chr = row["CHROM"]
7246            pos = row["POS"]
7247            ref = row["REF"]
7248            alt = row["ALT"]
7249
7250            # Find list of associated transcripts
7251            transcripts_list = list(
7252                polars_conn.execute(
7253                    f"""
7254                SELECT transcript
7255                FROM refseq_df
7256                WHERE CHROM='{chr}'
7257                AND POS={pos}
7258            """
7259                )["transcript"]
7260            )
7261
7262            # Full HGVS annotation in list
7263            hgvs_full_list = []
7264
7265            for transcript_name in transcripts_list:
7266
7267                # Transcript
7268                transcript = get_transcript(
7269                    transcripts=transcripts, transcript_name=transcript_name
7270                )
7271                # Exon
7272                if use_exon:
7273                    exon = transcript.find_exon_number(pos)
7274                else:
7275                    exon = None
7276                # Protein
7277                transcript_protein = None
7278                if use_protein or add_protein or full_format:
7279                    transcripts_protein = list(
7280                        polars_conn.execute(
7281                            f"""
7282                        SELECT protein
7283                        FROM refseqlink_df
7284                        WHERE transcript='{transcript_name}'
7285                        LIMIT 1
7286                    """
7287                        )["protein"]
7288                    )
7289                    if len(transcripts_protein):
7290                        transcript_protein = transcripts_protein[0]
7291
7292                # HGVS name
7293                hgvs_name = format_hgvs_name(
7294                    chr,
7295                    pos,
7296                    ref,
7297                    alt,
7298                    genome=genome,
7299                    transcript=transcript,
7300                    transcript_protein=transcript_protein,
7301                    exon=exon,
7302                    use_gene=use_gene,
7303                    use_protein=use_protein,
7304                    full_format=full_format,
7305                    use_version=use_version,
7306                    codon_type=codon_type,
7307                )
7308                hgvs_full_list.append(hgvs_name)
7309                if add_protein and not use_protein and not full_format:
7310                    hgvs_name = format_hgvs_name(
7311                        chr,
7312                        pos,
7313                        ref,
7314                        alt,
7315                        genome=genome,
7316                        transcript=transcript,
7317                        transcript_protein=transcript_protein,
7318                        exon=exon,
7319                        use_gene=use_gene,
7320                        use_protein=True,
7321                        full_format=False,
7322                        use_version=use_version,
7323                        codon_type=codon_type,
7324                    )
7325                    hgvs_full_list.append(hgvs_name)
7326
7327            # Create liste of HGVS annotations
7328            hgvs_full = ",".join(hgvs_full_list)
7329
7330            return hgvs_full
7331
7332        # Polars connexion
7333        polars_conn = pl.SQLContext(register_globals=True, eager=True)
7334
7335        # Config
7336        config = self.get_config()
7337
7338        # Databases
7339        # Genome
7340        databases_genomes_folders = (
7341            config.get("folders", {})
7342            .get("databases", {})
7343            .get("genomes", DEFAULT_GENOME_FOLDER)
7344        )
7345        databases_genome = (
7346            config.get("folders", {}).get("databases", {}).get("genomes", "")
7347        )
7348        # refseq database folder
7349        databases_refseq_folders = (
7350            config.get("folders", {})
7351            .get("databases", {})
7352            .get("refseq", DEFAULT_REFSEQ_FOLDER)
7353        )
7354        # refseq
7355        databases_refseq = config.get("databases", {}).get("refSeq", None)
7356        # refSeqLink
7357        databases_refseqlink = config.get("databases", {}).get("refSeqLink", None)
7358
7359        # Param
7360        param = self.get_param()
7361
7362        # Quick HGVS
7363        if "hgvs_options" in param and param.get("hgvs_options", ""):
7364            log.info(f"Quick HGVS Annotation:")
7365            if not param.get("hgvs", None):
7366                param["hgvs"] = {}
7367            for option in param.get("hgvs_options", "").split(","):
7368                option_var_val = option.split("=")
7369                option_var = option_var_val[0]
7370                if len(option_var_val) > 1:
7371                    option_val = option_var_val[1]
7372                else:
7373                    option_val = "True"
7374                if option_val.upper() in ["TRUE"]:
7375                    option_val = True
7376                elif option_val.upper() in ["FALSE"]:
7377                    option_val = False
7378                log.info(f"   {option_var}={option_val}")
7379                param["hgvs"][option_var] = option_val
7380
7381        # Check if HGVS annotation enabled
7382        if "hgvs" in param:
7383            log.info(f"HGVS Annotation... ")
7384            for hgvs_option in param.get("hgvs", {}):
7385                log.info(f"{hgvs_option}: {param.get('hgvs',{}).get(hgvs_option)}")
7386        else:
7387            return
7388
7389        # HGVS Param
7390        param_hgvs = param.get("hgvs", {})
7391        use_exon = param_hgvs.get("use_exon", False)
7392        use_gene = param_hgvs.get("use_gene", False)
7393        use_protein = param_hgvs.get("use_protein", False)
7394        add_protein = param_hgvs.get("add_protein", False)
7395        full_format = param_hgvs.get("full_format", False)
7396        use_version = param_hgvs.get("use_version", False)
7397        codon_type = param_hgvs.get("codon_type", "3")
7398
7399        # refSseq refSeqLink
7400        databases_refseq = param_hgvs.get("refseq", databases_refseq)
7401        databases_refseqlink = param_hgvs.get("refseqlink", databases_refseqlink)
7402
7403        # Assembly
7404        assembly = param.get("assembly", config.get("assembly", DEFAULT_ASSEMBLY))
7405
7406        # Genome
7407        genome_file = None
7408        if find_genome(databases_genome):
7409            genome_file = find_genome(databases_genome)
7410        else:
7411            genome_file = find_genome(
7412                genome_path=databases_genomes_folders, assembly=assembly
7413            )
7414        log.debug("Genome: " + str(genome_file))
7415
7416        # refSseq
7417        refseq_file = find_file_prefix(
7418            input_file=databases_refseq,
7419            prefix="ncbiRefSeq",
7420            folder=databases_refseq_folders,
7421            assembly=assembly,
7422        )
7423        log.debug("refSeq: " + str(refseq_file))
7424
7425        # refSeqLink
7426        refseqlink_file = find_file_prefix(
7427            input_file=databases_refseqlink,
7428            prefix="ncbiRefSeqLink",
7429            folder=databases_refseq_folders,
7430            assembly=assembly,
7431        )
7432        log.debug("refSeqLink: " + str(refseqlink_file))
7433
7434        # Threads
7435        if not threads:
7436            threads = self.get_threads()
7437        log.debug("Threads: " + str(threads))
7438
7439        # Variables
7440        table_variants = self.get_table_variants(clause="update")
7441
7442        # Get variants SNV and InDel only
7443        query_variants = f"""
7444            SELECT "#CHROM" AS CHROM, POS, REF, ALT
7445            FROM {table_variants}
7446            WHERE REF ~ '^[A-Za-z]+$' AND ALT ~ '^[A-Za-z]+$'
7447            """
7448        df_variants = self.get_query_to_df(query_variants)
7449
7450        # Added columns
7451        added_columns = []
7452
7453        # Add hgvs column in variants table
7454        hgvs_column_name = "hgvs_" + str(random.randrange(1000))
7455        added_column = self.add_column(
7456            table_variants, hgvs_column_name, "STRING", default_value=None
7457        )
7458        added_columns.append(added_column)
7459
7460        log.debug(f"refSeq loading...")
7461        # refSeq in duckDB
7462        refseq_table = get_refseq_table(
7463            conn=self.conn, refseq_table="refseq", refseq_file=refseq_file
7464        )
7465        # Loading all refSeq in Dataframe
7466        refseq_query = f"""
7467            SELECT df_variants.CHROM, df_variants.POS, {refseq_table}.name AS transcript
7468            FROM {refseq_table}
7469            JOIN df_variants ON (
7470                {refseq_table}.chrom = df_variants.CHROM
7471                AND {refseq_table}.txStart<=df_variants.POS
7472                AND {refseq_table}.txEnd>=df_variants.POS
7473            )
7474        """
7475        refseq_df = self.conn.query(refseq_query).pl()
7476
7477        if refseqlink_file:
7478            log.debug(f"refSeqLink loading...")
7479            # refSeqLink in duckDB
7480            refseqlink_table = get_refseq_table(
7481                conn=self.conn, refseq_table="refseqlink", refseq_file=refseqlink_file
7482            )
7483            # Loading all refSeqLink in Dataframe
7484            protacc_column = "protAcc_with_ver"
7485            mrnaacc_column = "mrnaAcc_with_ver"
7486            refseqlink_query = f"""
7487                SELECT {refseq_table}.chrom, {protacc_column} AS protein, {mrnaacc_column} AS transcript
7488                FROM {refseqlink_table} 
7489                JOIN {refseq_table} ON ({refseq_table}.name = {refseqlink_table}.mrnaAcc_with_ver)
7490                WHERE protAcc_without_ver IS NOT NULL
7491            """
7492            # Polars Dataframe
7493            refseqlink_df = self.conn.query(f"{refseqlink_query}").pl()
7494
7495        # Read RefSeq transcripts into a python dict/model.
7496        log.debug(f"Transcripts loading...")
7497        with tempfile.TemporaryDirectory() as tmpdir:
7498            transcripts_query = f"""
7499                COPY (
7500                    SELECT {refseq_table}.*
7501                    FROM {refseq_table}
7502                    JOIN df_variants ON (
7503                        {refseq_table}.chrom=df_variants.CHROM
7504                        AND {refseq_table}.txStart<=df_variants.POS
7505                        AND {refseq_table}.txEnd>=df_variants.POS
7506                    )
7507                )
7508                TO '{tmpdir}/transcript.tsv' (DELIMITER '\t');
7509            """
7510            self.conn.query(transcripts_query)
7511            with open(f"{tmpdir}/transcript.tsv") as infile:
7512                transcripts = read_transcripts(infile)
7513
7514        # Polars connexion
7515        polars_conn = pl.SQLContext(register_globals=True, eager=True)
7516
7517        log.debug("Genome loading...")
7518        # Read genome sequence using pyfaidx.
7519        genome = Fasta(genome_file)
7520
7521        log.debug("Start annotation HGVS...")
7522
7523        # Create
7524        # a Dask Dataframe from Pandas dataframe with partition as number of threads
7525        ddf = dd.from_pandas(df_variants, npartitions=threads)
7526
7527        # Use dask.dataframe.apply() to apply function on each partition
7528        ddf[hgvs_column_name] = ddf.map_partitions(partition_function)
7529
7530        # Convert Dask DataFrame to Pandas Dataframe
7531        df = ddf.compute()
7532
7533        # Convert Pandas dataframe to parquet (due to error in cast VARCHAR -> NULL ???)
7534        with tempfile.TemporaryDirectory() as tmpdir:
7535            df_parquet = os.path.join(tmpdir, "df.parquet")
7536            df.to_parquet(df_parquet)
7537
7538            # Update hgvs column
7539            update_variant_query = f"""
7540                UPDATE {table_variants}
7541                SET "{hgvs_column_name}"=df."{hgvs_column_name}"
7542                FROM read_parquet('{df_parquet}') as df
7543                WHERE variants."#CHROM" = df.CHROM
7544                AND variants.POS = df.POS
7545                AND variants.REF = df.REF
7546                AND variants.ALT = df.ALT
7547                AND df."{hgvs_column_name}" NOT IN ('') AND df."{hgvs_column_name}" NOT NULL
7548                """
7549            self.execute_query(update_variant_query)
7550
7551        # Update INFO column
7552        sql_query_update = f"""
7553            UPDATE {table_variants}
7554            SET INFO = 
7555                concat(
7556                    CASE 
7557                        WHEN INFO NOT IN ('','.')
7558                        THEN concat(INFO, ';')
7559                        ELSE ''
7560                    END,
7561                    'hgvs=',
7562                    {hgvs_column_name}
7563                )
7564            WHERE "{hgvs_column_name}" NOT IN ('') AND "{hgvs_column_name}" NOT NULL
7565            """
7566        self.execute_query(sql_query_update)
7567
7568        # Add header
7569        HGVS_INFOS = {
7570            "hgvs": {
7571                "ID": "hgvs",
7572                "Number": ".",
7573                "Type": "String",
7574                "Description": f"HGVS annotatation with HOWARD",
7575            }
7576        }
7577
7578        for field in HGVS_INFOS:
7579            field_ID = HGVS_INFOS[field]["ID"]
7580            field_description = HGVS_INFOS[field]["Description"]
7581            self.get_header().infos[field_ID] = vcf.parser._Info(
7582                field_ID,
7583                HGVS_INFOS[field]["Number"],
7584                HGVS_INFOS[field]["Type"],
7585                field_description,
7586                "unknown",
7587                "unknown",
7588                code_type_map[HGVS_INFOS[field]["Type"]],
7589            )
7590
7591        # Remove added columns
7592        for added_column in added_columns:
7593            self.drop_column(column=added_column)

The annotation_hgvs function performs HGVS annotation on a set of variants using genomic coordinates and alleles.

Parameters

threads: The threads parameter is an optional integer that specifies the number of threads to use for parallel processing. If no value is provided, it will default to the number of threads obtained from the get_threads() method

def get_operations_help( self, operations_config_dict: dict = {}, operations_config_file: str = None) -> list: View Source

7599    def get_operations_help(
7600        self, operations_config_dict: dict = {}, operations_config_file: str = None
7601    ) -> list:
7602
7603        # Init
7604        operations_help = []
7605
7606        # operations
7607        operations = self.get_config_json(
7608            name="calculations",
7609            config_dict=operations_config_dict,
7610            config_file=operations_config_file,
7611        )
7612        for op in operations:
7613            op_name = operations[op].get("name", op).upper()
7614            op_description = operations[op].get("description", op_name)
7615            op_available = operations[op].get("available", False)
7616            if op_available:
7617                operations_help.append(f"   {op_name}: {op_description}")
7618
7619        # Sort operations
7620        operations_help.sort()
7621
7622        # insert header
7623        operations_help.insert(0, "Available calculation operations:")
7624
7625        # Return
7626        return operations_help

def calculation( self, operations: dict = {}, operations_config_dict: dict = {}, operations_config_file: str = None) -> None: View Source

7628    def calculation(
7629        self,
7630        operations: dict = {},
7631        operations_config_dict: dict = {},
7632        operations_config_file: str = None,
7633    ) -> None:
7634        """
7635        It takes a list of operations, and for each operation, it checks if it's a python or sql
7636        operation, and then calls the appropriate function
7637
7638        param json example:
7639            "calculation": {
7640                "NOMEN": {
7641                    "options": {
7642                        "hgvs_field": "hgvs"
7643                    },
7644                "middle" : null
7645            }
7646        """
7647
7648        # Param
7649        param = self.get_param()
7650
7651        # operations config
7652        operations_config = self.get_config_json(
7653            name="calculations",
7654            config_dict=operations_config_dict,
7655            config_file=operations_config_file,
7656        )
7657
7658        # Upper keys
7659        operations_config = {k.upper(): v for k, v in operations_config.items()}
7660
7661        # Calculations
7662
7663        # Operations from param
7664        operations = param.get("calculation", {}).get("calculations", operations)
7665
7666        # Quick calculation - add
7667        if param.get("calculations", None):
7668            calculations_list = [
7669                value for value in param.get("calculations", "").split(",")
7670            ]
7671            log.info(f"Quick Calculations:")
7672            for calculation_key in calculations_list:
7673                log.info(f"   {calculation_key}")
7674            for calculation_operation in calculations_list:
7675                if calculation_operation.upper() not in operations:
7676                    operations[calculation_operation.upper()] = {}
7677                    add_value_into_dict(
7678                        dict_tree=param,
7679                        sections=[
7680                            "calculation",
7681                            "calculations",
7682                            calculation_operation.upper(),
7683                        ],
7684                        value={},
7685                    )
7686
7687        # Operations for calculation
7688        if not operations:
7689            operations = param.get("calculation", {}).get("calculations", {})
7690
7691        if operations:
7692            log.info(f"Calculations...")
7693
7694        # For each operations
7695        for operation_name in operations:
7696            operation_name = operation_name.upper()
7697            if operation_name not in [""]:
7698                if operation_name in operations_config:
7699                    log.info(f"Calculation '{operation_name}'")
7700                    operation = operations_config[operation_name]
7701                    operation_type = operation.get("type", "sql")
7702                    if operation_type == "python":
7703                        self.calculation_process_function(
7704                            operation=operation, operation_name=operation_name
7705                        )
7706                    elif operation_type == "sql":
7707                        self.calculation_process_sql(
7708                            operation=operation, operation_name=operation_name
7709                        )
7710                    else:
7711                        log.error(
7712                            f"Operations config: Type '{operation_type}' NOT available"
7713                        )
7714                        raise ValueError(
7715                            f"Operations config: Type '{operation_type}' NOT available"
7716                        )
7717                else:
7718                    log.error(
7719                        f"Operations config: Calculation '{operation_name}' NOT available"
7720                    )
7721                    raise ValueError(
7722                        f"Operations config: Calculation '{operation_name}' NOT available"
7723                    )
7724
7725        # Explode INFOS fields into table fields
7726        if self.get_explode_infos():
7727            self.explode_infos(
7728                prefix=self.get_explode_infos_prefix(),
7729                fields=self.get_explode_infos_fields(),
7730                force=True,
7731            )

It takes a list of operations, and for each operation, it checks if it's a python or sql operation, and then calls the appropriate function

param json example: "calculation": { "NOMEN": { "options": { "hgvs_field": "hgvs" }, "middle" : null }

def calculation_process_sql(self, operation: dict, operation_name: str = 'unknown') -> None: View Source

7733    def calculation_process_sql(
7734        self, operation: dict, operation_name: str = "unknown"
7735    ) -> None:
7736        """
7737        The `calculation_process_sql` function takes in a mathematical operation as a string and
7738        performs the operation, updating the specified table with the result.
7739
7740        :param operation: The `operation` parameter is a dictionary that contains information about the
7741        mathematical operation to be performed. It includes the following keys:
7742        :type operation: dict
7743        :param operation_name: The `operation_name` parameter is a string that represents the name of
7744        the mathematical operation being performed. It is used for logging and error handling purposes,
7745        defaults to unknown
7746        :type operation_name: str (optional)
7747        """
7748
7749        # table variants
7750        table_variants = self.get_table_variants(clause="alter")
7751
7752        # Operation infos
7753        operation_name = operation.get("name", "unknown")
7754        log.debug(f"process sql {operation_name}")
7755        output_column_name = operation.get("output_column_name", operation_name)
7756        output_column_type = operation.get("output_column_type", "String")
7757        prefix = operation.get("explode_infos_prefix", "")
7758        output_column_type_sql = code_type_map_to_sql.get(output_column_type, "VARCHAR")
7759        output_column_description = operation.get(
7760            "output_column_description", f"{operation_name} operation"
7761        )
7762        operation_query = operation.get("operation_query", None)
7763        if isinstance(operation_query, list):
7764            operation_query = " ".join(operation_query)
7765        operation_info_fields = operation.get("info_fields", [])
7766        operation_info_fields_check = operation.get("info_fields_check", False)
7767        operation_info = operation.get("operation_info", True)
7768
7769        if operation_query:
7770
7771            # Info fields check
7772            operation_info_fields_check_result = True
7773            if operation_info_fields_check:
7774                header_infos = self.get_header().infos
7775                for info_field in operation_info_fields:
7776                    operation_info_fields_check_result = (
7777                        operation_info_fields_check_result
7778                        and info_field in header_infos
7779                    )
7780
7781            # If info fields available
7782            if operation_info_fields_check_result:
7783
7784                # Added_columns
7785                added_columns = []
7786
7787                # Create VCF header field
7788                vcf_reader = self.get_header()
7789                vcf_reader.infos[output_column_name] = vcf.parser._Info(
7790                    output_column_name,
7791                    ".",
7792                    output_column_type,
7793                    output_column_description,
7794                    "howard calculation",
7795                    "0",
7796                    self.code_type_map.get(output_column_type),
7797                )
7798
7799                # Explode infos if needed
7800                log.debug(f"calculation_process_sql prefix {prefix}")
7801                added_columns += self.explode_infos(
7802                    prefix=prefix,
7803                    fields=[output_column_name] + operation_info_fields,
7804                    force=True,
7805                )
7806
7807                # Create column
7808                added_column = self.add_column(
7809                    table_name=table_variants,
7810                    column_name=prefix + output_column_name,
7811                    column_type=output_column_type_sql,
7812                    default_value="null",
7813                )
7814                added_columns.append(added_column)
7815
7816                # Operation calculation
7817                try:
7818
7819                    # Query to update calculation column
7820                    sql_update = f"""
7821                        UPDATE {table_variants}
7822                        SET "{prefix}{output_column_name}" = ({operation_query})
7823                    """
7824                    self.conn.execute(sql_update)
7825
7826                    # Add to INFO
7827                    if operation_info:
7828                        sql_update_info = f"""
7829                            UPDATE {table_variants}
7830                            SET "INFO" =
7831                                concat(
7832                                    CASE
7833                                        WHEN "INFO" IS NOT NULL
7834                                        THEN concat("INFO", ';')
7835                                        ELSE ''
7836                                    END,
7837                                    '{output_column_name}=',
7838                                    "{prefix}{output_column_name}"
7839                                )
7840                            WHERE "{prefix}{output_column_name}" IS NOT NULL AND "{prefix}{output_column_name}" NOT IN ('')
7841                        """
7842                        self.conn.execute(sql_update_info)
7843
7844                except:
7845                    log.error(
7846                        f"Operations config: Calculation '{operation_name}' query failed"
7847                    )
7848                    raise ValueError(
7849                        f"Operations config: Calculation '{operation_name}' query failed"
7850                    )
7851
7852                # Remove added columns
7853                for added_column in added_columns:
7854                    log.debug(f"added_column: {added_column}")
7855                    self.drop_column(column=added_column)
7856
7857            else:
7858                log.error(
7859                    f"Operations config: Calculation '{operation_name}' DOES NOT contain all mandatory fields {operation_info_fields}"
7860                )
7861                raise ValueError(
7862                    f"Operations config: Calculation '{operation_name}' DOES NOT contain all mandatory fields {operation_info_fields}"
7863                )
7864
7865        else:
7866            log.error(
7867                f"Operations config: Calculation '{operation_name}' query NOT defined"
7868            )
7869            raise ValueError(
7870                f"Operations config: Calculation '{operation_name}' query NOT defined"
7871            )

The calculation_process_sql function takes in a mathematical operation as a string and performs the operation, updating the specified table with the result.

Parameters

operation: The operation parameter is a dictionary that contains information about the mathematical operation to be performed. It includes the following keys:
operation_name: The operation_name parameter is a string that represents the name of the mathematical operation being performed. It is used for logging and error handling purposes, defaults to unknown

def calculation_process_function(self, operation: dict, operation_name: str = 'unknown') -> None: View Source

7873    def calculation_process_function(
7874        self, operation: dict, operation_name: str = "unknown"
7875    ) -> None:
7876        """
7877        The `calculation_process_function` takes in an operation dictionary and performs the specified
7878        function with the given parameters.
7879
7880        :param operation: The `operation` parameter is a dictionary that contains information about the
7881        operation to be performed. It has the following keys:
7882        :type operation: dict
7883        :param operation_name: The `operation_name` parameter is a string that represents the name of
7884        the operation being performed. It is used for logging purposes, defaults to unknown
7885        :type operation_name: str (optional)
7886        """
7887
7888        operation_name = operation["name"]
7889        log.debug(f"process sql {operation_name}")
7890        function_name = operation["function_name"]
7891        function_params = operation["function_params"]
7892        getattr(self, function_name)(*function_params)

The calculation_process_function takes in an operation dictionary and performs the specified function with the given parameters.

Parameters

operation: The operation parameter is a dictionary that contains information about the operation to be performed. It has the following keys:
operation_name: The operation_name parameter is a string that represents the name of the operation being performed. It is used for logging purposes, defaults to unknown

def calculation_variant_id(self) -> None: View Source

7894    def calculation_variant_id(self) -> None:
7895        """
7896        The function `calculation_variant_id` adds a variant ID annotation to a VCF file header and
7897        updates the INFO field of a variants table with the variant ID.
7898        """
7899
7900        # variant_id annotation field
7901        variant_id_tag = self.get_variant_id_column()
7902        added_columns = [variant_id_tag]
7903
7904        # variant_id hgvs tags"
7905        vcf_infos_tags = {
7906            variant_id_tag: "howard variant ID annotation",
7907        }
7908
7909        # Variants table
7910        table_variants = self.get_table_variants()
7911
7912        # Header
7913        vcf_reader = self.get_header()
7914
7915        # Add variant_id to header
7916        vcf_reader.infos[variant_id_tag] = vcf.parser._Info(
7917            variant_id_tag,
7918            ".",
7919            "String",
7920            vcf_infos_tags.get(variant_id_tag, "howard variant ID annotation"),
7921            "howard calculation",
7922            "0",
7923            self.code_type_map.get("String"),
7924        )
7925
7926        # Update
7927        sql_update = f"""
7928            UPDATE {table_variants}
7929            SET "INFO" = 
7930                concat(
7931                    CASE
7932                        WHEN "INFO" IS NULL OR "INFO" IN ('','.')
7933                        THEN ''
7934                        ELSE concat("INFO", ';')
7935                    END,
7936                    '{variant_id_tag}=',
7937                    "{variant_id_tag}"
7938                )
7939        """
7940        self.conn.execute(sql_update)
7941
7942        # Remove added columns
7943        for added_column in added_columns:
7944            self.drop_column(column=added_column)

The function calculation_variant_id adds a variant ID annotation to a VCF file header and updates the INFO field of a variants table with the variant ID.

def calculation_extract_snpeff_hgvs( self, snpeff_hgvs: str = 'snpeff_hgvs', snpeff_field: str = 'ANN') -> None: View Source

7946    def calculation_extract_snpeff_hgvs(
7947        self,
7948        snpeff_hgvs: str = "snpeff_hgvs",
7949        snpeff_field: str = "ANN",
7950    ) -> None:
7951        """
7952        The function `calculation_extract_snpeff_hgvs` extracts HGVS nomenclatures from the SnpEff
7953        annotation field in a VCF file and adds them as a new column in the variants table.
7954
7955        :param snpeff_hgvs: The `snpeff_hgvs` parameter in the `calculation_extract_snpeff_hgvs`
7956        function is used to specify the name of the column that will store the HGVS nomenclatures
7957        extracted from the SnpEff annotation field in a VCF file. This parameter allows you, defaults to
7958        snpeff_hgvs
7959        :type snpeff_hgvs: str (optional)
7960        :param snpeff_field: The `snpeff_field` parameter in the `calculation_extract_snpeff_hgvs`
7961        function represents the field in the VCF file that contains SnpEff annotations. This field is
7962        used to extract HGVS nomenclatures from the SnpEff annotation field and add them as a, defaults
7963        to ANN
7964        :type snpeff_field: str (optional)
7965        """
7966
7967        # Snpeff hgvs tags
7968        vcf_infos_tags = {
7969            snpeff_hgvs: "HGVS nomenclatures from snpEff annotation",
7970        }
7971
7972        # Prefix
7973        prefix = self.get_explode_infos_prefix()
7974        if prefix:
7975            prefix = "INFO/"
7976
7977        # snpEff fields
7978        speff_ann_infos = prefix + snpeff_field
7979        speff_hgvs_infos = prefix + snpeff_hgvs
7980
7981        # Variants table
7982        table_variants = self.get_table_variants()
7983
7984        # Header
7985        vcf_reader = self.get_header()
7986
7987        # Add columns
7988        added_columns = []
7989
7990        # Explode HGVS field in column
7991        added_columns += self.explode_infos(fields=[snpeff_field])
7992
7993        if snpeff_field in vcf_reader.infos:
7994
7995            log.debug(vcf_reader.infos[snpeff_field])
7996
7997            # Extract ANN header
7998            ann_description = vcf_reader.infos[snpeff_field].desc
7999            pattern = r"'(.+?)'"
8000            match = re.search(pattern, ann_description)
8001            if match:
8002                ann_header_match = match.group(1).split(" | ")
8003                ann_header_desc = {}
8004                for i in range(len(ann_header_match)):
8005                    ann_header_info = "".join(
8006                        char for char in ann_header_match[i] if char.isalnum()
8007                    )
8008                    ann_header_desc[ann_header_info] = ann_header_match[i]
8009                if not ann_header_desc:
8010                    raise ValueError("Invalid header description format")
8011            else:
8012                raise ValueError("Invalid header description format")
8013
8014            # Create variant id
8015            variant_id_column = self.get_variant_id_column()
8016            added_columns += [variant_id_column]
8017
8018            # Create dataframe
8019            dataframe_snpeff_hgvs = self.get_query_to_df(
8020                f""" SELECT "{variant_id_column}", "{speff_ann_infos}" FROM {table_variants} """
8021            )
8022
8023            # Create main NOMEN column
8024            dataframe_snpeff_hgvs[speff_hgvs_infos] = dataframe_snpeff_hgvs[
8025                speff_ann_infos
8026            ].apply(
8027                lambda x: extract_snpeff_hgvs(
8028                    str(x), header=list(ann_header_desc.values())
8029                )
8030            )
8031
8032            # Add snpeff_hgvs to header
8033            vcf_reader.infos[snpeff_hgvs] = vcf.parser._Info(
8034                snpeff_hgvs,
8035                ".",
8036                "String",
8037                vcf_infos_tags.get(snpeff_hgvs, "snpEff hgvs annotations"),
8038                "howard calculation",
8039                "0",
8040                self.code_type_map.get("String"),
8041            )
8042
8043            # Update
8044            sql_update = f"""
8045                UPDATE variants
8046                SET "INFO" = 
8047                    concat(
8048                        CASE
8049                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8050                            THEN ''
8051                            ELSE concat("INFO", ';')
8052                        END,
8053                        CASE 
8054                            WHEN dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT IN ('','.','NaN')
8055                            AND dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT NULL
8056                            THEN concat(
8057                                    '{snpeff_hgvs}=',
8058                                    dataframe_snpeff_hgvs."{speff_hgvs_infos}"
8059                                )
8060                            ELSE ''
8061                        END
8062                    )
8063                FROM dataframe_snpeff_hgvs
8064                WHERE {table_variants}."{variant_id_column}" = dataframe_snpeff_hgvs."{variant_id_column}"
8065
8066            """
8067            self.conn.execute(sql_update)
8068
8069            # Delete dataframe
8070            del dataframe_snpeff_hgvs
8071            gc.collect()
8072
8073        else:
8074
8075            log.warning(
8076                "No snpEff annotation. Please Anotate with snpEff before use this calculation option"
8077            )
8078
8079        # Remove added columns
8080        for added_column in added_columns:
8081            self.drop_column(column=added_column)

The function calculation_extract_snpeff_hgvs extracts HGVS nomenclatures from the SnpEff annotation field in a VCF file and adds them as a new column in the variants table.

Parameters

snpeff_hgvs: The snpeff_hgvs parameter in the calculation_extract_snpeff_hgvs function is used to specify the name of the column that will store the HGVS nomenclatures extracted from the SnpEff annotation field in a VCF file. This parameter allows you, defaults to snpeff_hgvs
snpeff_field: The snpeff_field parameter in the calculation_extract_snpeff_hgvs function represents the field in the VCF file that contains SnpEff annotations. This field is used to extract HGVS nomenclatures from the SnpEff annotation field and add them as a, defaults to ANN

def calculation_snpeff_ann_explode( self, uniquify: bool = True, output_format: str = 'fields', output_prefix: str = 'snpeff_', snpeff_field: str = 'ANN') -> None: View Source

8083    def calculation_snpeff_ann_explode(
8084        self,
8085        uniquify: bool = True,
8086        output_format: str = "fields",
8087        output_prefix: str = "snpeff_",
8088        snpeff_field: str = "ANN",
8089    ) -> None:
8090        """
8091        The `calculation_snpeff_ann_explode` function processes SnpEff annotations in a VCF file by
8092        exploding the HGVS field and updating variant information accordingly.
8093
8094        :param uniquify: The `uniquify` parameter in the `calculation_snpeff_ann_explode` method is a
8095        boolean flag that determines whether the output should be uniquified or not. When set to `True`,
8096        it indicates that the output should be unique, meaning that duplicate entries should be removed,
8097        defaults to True
8098        :type uniquify: bool (optional)
8099        :param output_format: The `output_format` parameter in the `calculation_snpeff_ann_explode`
8100        function specifies the format in which the output annotations will be generated. It has a
8101        default value of "fields". You can also set it to "JSON" to output the annotations in JSON
8102        format, defaults to fields
8103        :type output_format: str (optional)
8104        :param output_prefix: The `output_prefix` parameter in the `calculation_snpeff_ann_explode`
8105        method is used to specify the prefix that will be added to the output annotations generated
8106        during the calculation process. This prefix helps to differentiate the newly added annotations
8107        from existing ones in the output data. By default, the, defaults to ANN_
8108        :type output_prefix: str (optional)
8109        :param snpeff_field: The `snpeff_field` parameter in the `calculation_snpeff_ann_explode`
8110        function is used to specify the field in the VCF file that contains SnpEff annotations. This
8111        field will be processed to explode the HGVS annotations and update the variant information
8112        accordingly, defaults to ANN
8113        :type snpeff_field: str (optional)
8114        """
8115
8116        # SnpEff annotation field
8117        snpeff_hgvs = "snpeff_ann_explode"
8118
8119        # Snpeff hgvs tags
8120        vcf_infos_tags = {
8121            snpeff_hgvs: "Explode snpEff annotations",
8122        }
8123
8124        # Prefix
8125        prefix = self.get_explode_infos_prefix()
8126        if prefix:
8127            prefix = "INFO/"
8128
8129        # snpEff fields
8130        speff_ann_infos = prefix + snpeff_field
8131        speff_hgvs_infos = prefix + snpeff_hgvs
8132
8133        # Variants table
8134        table_variants = self.get_table_variants()
8135
8136        # Header
8137        vcf_reader = self.get_header()
8138
8139        # Add columns
8140        added_columns = []
8141
8142        # Explode HGVS field in column
8143        added_columns += self.explode_infos(fields=[snpeff_field])
8144        log.debug(f"snpeff_field={snpeff_field}")
8145        log.debug(f"added_columns={added_columns}")
8146
8147        if snpeff_field in vcf_reader.infos:
8148
8149            # Extract ANN header
8150            ann_description = vcf_reader.infos[snpeff_field].desc
8151            pattern = r"'(.+?)'"
8152            match = re.search(pattern, ann_description)
8153            if match:
8154                ann_header_match = match.group(1).split(" | ")
8155                ann_header = []
8156                ann_header_desc = {}
8157                for i in range(len(ann_header_match)):
8158                    ann_header_info = "".join(
8159                        char for char in ann_header_match[i] if char.isalnum()
8160                    )
8161                    ann_header.append(ann_header_info)
8162                    ann_header_desc[ann_header_info] = ann_header_match[i]
8163                if not ann_header_desc:
8164                    raise ValueError("Invalid header description format")
8165            else:
8166                raise ValueError("Invalid header description format")
8167
8168            # Create variant id
8169            variant_id_column = self.get_variant_id_column()
8170            added_columns += [variant_id_column]
8171
8172            # Create dataframe
8173            dataframe_snpeff_hgvs = self.get_query_to_df(
8174                f""" SELECT "{variant_id_column}", "{speff_ann_infos}" FROM {table_variants} """
8175            )
8176
8177            # Create snpEff columns
8178            dataframe_snpeff_hgvs[speff_hgvs_infos] = dataframe_snpeff_hgvs[
8179                speff_ann_infos
8180            ].apply(
8181                lambda x: explode_snpeff_ann(
8182                    str(x),
8183                    uniquify=uniquify,
8184                    output_format=output_format,
8185                    prefix=output_prefix,
8186                    header=list(ann_header_desc.values()),
8187                )
8188            )
8189
8190            # Header
8191            ann_annotations_prefix = ""
8192            if output_format.upper() in ["JSON"]:
8193                ann_annotations_prefix = f"{output_prefix}="
8194                vcf_reader.infos[output_prefix] = vcf.parser._Info(
8195                    output_prefix,
8196                    ".",
8197                    "String",
8198                    vcf_infos_tags.get(snpeff_hgvs, "snpEff annotations")
8199                    + " - JSON format",
8200                    "howard calculation",
8201                    "0",
8202                    self.code_type_map.get("String"),
8203                )
8204            else:
8205                for ann_annotation in ann_header:
8206                    ann_annotation_id = f"{output_prefix}{ann_annotation}"
8207                    vcf_reader.infos[ann_annotation_id] = vcf.parser._Info(
8208                        ann_annotation_id,
8209                        ".",
8210                        "String",
8211                        vcf_infos_tags.get(snpeff_hgvs, "snpEff annotations")
8212                        + f" - '{ann_header_desc[ann_annotation]}' annotation",
8213                        "howard calculation",
8214                        "0",
8215                        self.code_type_map.get("String"),
8216                    )
8217
8218            # Update
8219            sql_update = f"""
8220                UPDATE variants
8221                SET "INFO" = 
8222                    concat(
8223                        CASE
8224                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8225                            THEN ''
8226                            ELSE concat("INFO", ';')
8227                        END,
8228                        CASE 
8229                            WHEN dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT IN ('','.','NaN')
8230                                AND dataframe_snpeff_hgvs."{speff_hgvs_infos}" NOT NULL
8231                            THEN concat(
8232                                '{ann_annotations_prefix}',
8233                                dataframe_snpeff_hgvs."{speff_hgvs_infos}"
8234                                )
8235                            ELSE ''
8236                        END
8237                    )
8238                FROM dataframe_snpeff_hgvs
8239                WHERE {table_variants}."{variant_id_column}" = dataframe_snpeff_hgvs."{variant_id_column}"
8240
8241            """
8242            self.conn.execute(sql_update)
8243
8244            # Delete dataframe
8245            del dataframe_snpeff_hgvs
8246            gc.collect()
8247
8248        else:
8249
8250            log.warning(
8251                "No snpEff annotation. Please Anotate with snpEff before use this calculation option"
8252            )
8253
8254        # Remove added columns
8255        for added_column in added_columns:
8256            self.drop_column(column=added_column)

The calculation_snpeff_ann_explode function processes SnpEff annotations in a VCF file by exploding the HGVS field and updating variant information accordingly.

Parameters

uniquify: The uniquify parameter in the calculation_snpeff_ann_explode method is a boolean flag that determines whether the output should be uniquified or not. When set to True, it indicates that the output should be unique, meaning that duplicate entries should be removed, defaults to True
output_format: The output_format parameter in the calculation_snpeff_ann_explode function specifies the format in which the output annotations will be generated. It has a default value of "fields". You can also set it to "JSON" to output the annotations in JSON format, defaults to fields
output_prefix: The output_prefix parameter in the calculation_snpeff_ann_explode method is used to specify the prefix that will be added to the output annotations generated during the calculation process. This prefix helps to differentiate the newly added annotations from existing ones in the output data. By default, the, defaults to ANN_
snpeff_field: The snpeff_field parameter in the calculation_snpeff_ann_explode function is used to specify the field in the VCF file that contains SnpEff annotations. This field will be processed to explode the HGVS annotations and update the variant information accordingly, defaults to ANN

def calculation_extract_nomen(self) -> None: View Source

8258    def calculation_extract_nomen(self) -> None:
8259        """
8260        This function extracts the HGVS nomenclature from the calculation/identification of NOMEN.
8261        """
8262
8263        # NOMEN field
8264        field_nomen_dict = "NOMEN_DICT"
8265
8266        # NOMEN structure
8267        nomen_dict = {
8268            "NOMEN": "NOMEN hgvs nomenclature considered as reference hgvs (official transcript, first otherwise)",
8269            "CNOMEN": "CNOMEN hgvs nomenclature at DNA level related to a transcript (TNOMEN)",
8270            "RNOMEN": "RNOMEN hgvs nomenclature at RNA level related to a transcript (TNOMEN)",
8271            "NNOMEN": "NNOMEN hgvs nomenclature for non-coding variant",
8272            "PNOMEN": "PNOMEN hgvs nomenclature at Protein level related to a transcript (TNOMEN)",
8273            "TVNOMEN": "TVNOMEN hgvs transcript with version (if any) used (e.g. for CNOMEN and PNOMEN)",
8274            "TNOMEN": "TNOMEN hgvs transcript used (e.g. for CNOMEN and PNOMEN)",
8275            "VNOMEN": "VNOMEN hgvs transcript version used (e.g. for CNOMEN and PNOMEN)",
8276            "ENOMEN": "ENOMEN hgvs exon nomenclature related to a transcript (TNOMEN)",
8277            "GNOMEN": "GNOMEN hgvs gene nomenclature related to a transcript (TNOMEN)",
8278        }
8279
8280        # Param
8281        param = self.get_param()
8282
8283        # Prefix
8284        prefix = self.get_explode_infos_prefix()
8285
8286        # Header
8287        vcf_reader = self.get_header()
8288
8289        # Get HGVS field
8290        hgvs_field = (
8291            param.get("calculation", {})
8292            .get("calculations", {})
8293            .get("NOMEN", {})
8294            .get("options", {})
8295            .get("hgvs_field", "hgvs")
8296        )
8297
8298        # Get transcripts
8299        transcripts_file = (
8300            param.get("calculation", {})
8301            .get("calculations", {})
8302            .get("NOMEN", {})
8303            .get("options", {})
8304            .get("transcripts", None)
8305        )
8306        transcripts_file = full_path(transcripts_file)
8307        transcripts = []
8308        if transcripts_file:
8309            if os.path.exists(transcripts_file):
8310                transcripts_dataframe = transcripts_file_to_df(transcripts_file)
8311                transcripts = transcripts_dataframe.iloc[:, 0].tolist()
8312            else:
8313                log.error(f"Transcript file '{transcripts_file}' does NOT exist")
8314                raise ValueError(f"Transcript file '{transcripts_file}' does NOT exist")
8315
8316        # Added columns
8317        added_columns = []
8318
8319        # Explode HGVS field in column
8320        added_columns += self.explode_infos(fields=[hgvs_field])
8321
8322        # extra infos
8323        extra_infos = self.get_extra_infos()
8324        extra_field = prefix + hgvs_field
8325
8326        if extra_field in extra_infos:
8327
8328            # Create dataframe
8329            dataframe_hgvs = self.get_query_to_df(
8330                f""" SELECT "#CHROM", "POS", "REF", "ALT", "{extra_field}" FROM variants """
8331            )
8332
8333            # Create main NOMEN column
8334            dataframe_hgvs[field_nomen_dict] = dataframe_hgvs[extra_field].apply(
8335                lambda x: find_nomen(str(x), transcripts=transcripts)
8336            )
8337
8338            # Explode NOMEN Structure and create SQL set for update
8339            sql_nomen_fields = []
8340            for nomen_field in nomen_dict:
8341
8342                # Explode each field into a column
8343                dataframe_hgvs[nomen_field] = dataframe_hgvs[field_nomen_dict].apply(
8344                    lambda x: dict(x).get(nomen_field, "")
8345                )
8346
8347                # Create VCF header field
8348                vcf_reader.infos[nomen_field] = vcf.parser._Info(
8349                    nomen_field,
8350                    ".",
8351                    "String",
8352                    nomen_dict.get(nomen_field, "howard calculation NOMEN"),
8353                    "howard calculation",
8354                    "0",
8355                    self.code_type_map.get("String"),
8356                )
8357                sql_nomen_fields.append(
8358                    f"""
8359                        CASE 
8360                            WHEN dataframe_hgvs."{nomen_field}" NOT NULL AND dataframe_hgvs."{nomen_field}" NOT IN ('')
8361                            THEN concat(
8362                                    ';{nomen_field}=',
8363                                    dataframe_hgvs."{nomen_field}"
8364                                )
8365                            ELSE ''
8366                        END
8367                    """
8368                )
8369
8370            # SQL set for update
8371            sql_nomen_fields_set = ", ".join(sql_nomen_fields)
8372
8373            # Update
8374            sql_update = f"""
8375                UPDATE variants
8376                SET "INFO" = 
8377                    concat(
8378                        CASE
8379                            WHEN "INFO" IS NULL
8380                            THEN ''
8381                            ELSE "INFO"
8382                        END,
8383                        {sql_nomen_fields_set}
8384                    )
8385                FROM dataframe_hgvs
8386                WHERE variants."#CHROM" = dataframe_hgvs."#CHROM"
8387                    AND variants."POS" = dataframe_hgvs."POS" 
8388                    AND variants."REF" = dataframe_hgvs."REF"
8389                    AND variants."ALT" = dataframe_hgvs."ALT"
8390            """
8391            self.conn.execute(sql_update)
8392
8393            # Delete dataframe
8394            del dataframe_hgvs
8395            gc.collect()
8396
8397        # Remove added columns
8398        for added_column in added_columns:
8399            self.drop_column(column=added_column)

This function extracts the HGVS nomenclature from the calculation/identification of NOMEN.

def calculation_find_by_pipeline(self, tag: str = 'findbypipeline') -> None: View Source

8401    def calculation_find_by_pipeline(self, tag: str = "findbypipeline") -> None:
8402        """
8403        The function `calculation_find_by_pipeline` performs a calculation to find the number of
8404        pipeline/sample for a variant and updates the variant information in a VCF file.
8405
8406        :param tag: The `tag` parameter is a string that represents the annotation field for the
8407        "findbypipeline" information in the VCF file. It is used to create the annotation field in the
8408        VCF header and to update the corresponding field in the variants table, defaults to
8409        findbypipeline
8410        :type tag: str (optional)
8411        """
8412
8413        # if FORMAT and samples
8414        if (
8415            "FORMAT" in self.get_header_columns_as_list()
8416            and self.get_header_sample_list()
8417        ):
8418
8419            # findbypipeline annotation field
8420            findbypipeline_tag = tag
8421
8422            # VCF infos tags
8423            vcf_infos_tags = {
8424                findbypipeline_tag: f"Number of pipeline/sample for a variant ({findbypipeline_tag})",
8425            }
8426
8427            # Prefix
8428            prefix = self.get_explode_infos_prefix()
8429
8430            # Field
8431            findbypipeline_infos = prefix + findbypipeline_tag
8432
8433            # Variants table
8434            table_variants = self.get_table_variants()
8435
8436            # Header
8437            vcf_reader = self.get_header()
8438
8439            # Create variant id
8440            variant_id_column = self.get_variant_id_column()
8441            added_columns = [variant_id_column]
8442
8443            # variant_id, FORMAT and samples
8444            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
8445                self.get_header_sample_list()
8446            )
8447
8448            # Create dataframe
8449            dataframe_findbypipeline = self.get_query_to_df(
8450                f""" SELECT {samples_fields} FROM {table_variants} """
8451            )
8452
8453            # Create findbypipeline column
8454            dataframe_findbypipeline[findbypipeline_infos] = (
8455                dataframe_findbypipeline.apply(
8456                    lambda row: findbypipeline(
8457                        row, samples=self.get_header_sample_list()
8458                    ),
8459                    axis=1,
8460                )
8461            )
8462
8463            # Add snpeff_hgvs to header
8464            vcf_reader.infos[findbypipeline_tag] = vcf.parser._Info(
8465                findbypipeline_tag,
8466                ".",
8467                "String",
8468                vcf_infos_tags.get(findbypipeline_tag, "Find in pipeline/sample"),
8469                "howard calculation",
8470                "0",
8471                self.code_type_map.get("String"),
8472            )
8473
8474            # Update
8475            sql_update = f"""
8476                UPDATE variants
8477                SET "INFO" = 
8478                    concat(
8479                        CASE
8480                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8481                            THEN ''
8482                            ELSE concat("INFO", ';')
8483                        END,
8484                        CASE 
8485                            WHEN dataframe_findbypipeline."{findbypipeline_infos}" NOT IN ('','.')
8486                                AND dataframe_findbypipeline."{findbypipeline_infos}" NOT NULL
8487                            THEN concat(
8488                                    '{findbypipeline_tag}=',
8489                                    dataframe_findbypipeline."{findbypipeline_infos}"
8490                                )
8491                            ELSE ''
8492                        END
8493                    )
8494                FROM dataframe_findbypipeline
8495                WHERE variants."{variant_id_column}" = dataframe_findbypipeline."{variant_id_column}"
8496            """
8497            self.conn.execute(sql_update)
8498
8499            # Remove added columns
8500            for added_column in added_columns:
8501                self.drop_column(column=added_column)
8502
8503            # Delete dataframe
8504            del dataframe_findbypipeline
8505            gc.collect()

The function calculation_find_by_pipeline performs a calculation to find the number of pipeline/sample for a variant and updates the variant information in a VCF file.

Parameters

tag: The tag parameter is a string that represents the annotation field for the "findbypipeline" information in the VCF file. It is used to create the annotation field in the VCF header and to update the corresponding field in the variants table, defaults to findbypipeline

def calculation_genotype_concordance(self) -> None: View Source

8507    def calculation_genotype_concordance(self) -> None:
8508        """
8509        The function `calculation_genotype_concordance` calculates the genotype concordance for
8510        multi-caller VCF files and updates the variant information in the database.
8511        """
8512
8513        # if FORMAT and samples
8514        if (
8515            "FORMAT" in self.get_header_columns_as_list()
8516            and self.get_header_sample_list()
8517        ):
8518
8519            # genotypeconcordance annotation field
8520            genotypeconcordance_tag = "genotypeconcordance"
8521
8522            # VCF infos tags
8523            vcf_infos_tags = {
8524                genotypeconcordance_tag: "Concordance of genotype for multi caller VCF",
8525            }
8526
8527            # Prefix
8528            prefix = self.get_explode_infos_prefix()
8529
8530            # Field
8531            genotypeconcordance_infos = prefix + genotypeconcordance_tag
8532
8533            # Variants table
8534            table_variants = self.get_table_variants()
8535
8536            # Header
8537            vcf_reader = self.get_header()
8538
8539            # Create variant id
8540            variant_id_column = self.get_variant_id_column()
8541            added_columns = [variant_id_column]
8542
8543            # variant_id, FORMAT and samples
8544            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
8545                self.get_header_sample_list()
8546            )
8547
8548            # Create dataframe
8549            dataframe_genotypeconcordance = self.get_query_to_df(
8550                f""" SELECT {samples_fields} FROM {table_variants} """
8551            )
8552
8553            # Create genotypeconcordance column
8554            dataframe_genotypeconcordance[genotypeconcordance_infos] = (
8555                dataframe_genotypeconcordance.apply(
8556                    lambda row: genotypeconcordance(
8557                        row, samples=self.get_header_sample_list()
8558                    ),
8559                    axis=1,
8560                )
8561            )
8562
8563            # Add genotypeconcordance to header
8564            vcf_reader.infos[genotypeconcordance_tag] = vcf.parser._Info(
8565                genotypeconcordance_tag,
8566                ".",
8567                "String",
8568                vcf_infos_tags.get(genotypeconcordance_tag, "snpEff hgvs annotations"),
8569                "howard calculation",
8570                "0",
8571                self.code_type_map.get("String"),
8572            )
8573
8574            # Update
8575            sql_update = f"""
8576                UPDATE variants
8577                SET "INFO" = 
8578                    concat(
8579                        CASE
8580                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8581                            THEN ''
8582                            ELSE concat("INFO", ';')
8583                        END,
8584                        CASE
8585                            WHEN dataframe_genotypeconcordance."{genotypeconcordance_infos}" NOT IN ('','.')
8586                                AND dataframe_genotypeconcordance."{genotypeconcordance_infos}" NOT NULL
8587                            THEN concat(
8588                                    '{genotypeconcordance_tag}=',
8589                                    dataframe_genotypeconcordance."{genotypeconcordance_infos}"
8590                                )
8591                            ELSE ''
8592                        END
8593                    )
8594                FROM dataframe_genotypeconcordance
8595                WHERE variants."{variant_id_column}" = dataframe_genotypeconcordance."{variant_id_column}"
8596            """
8597            self.conn.execute(sql_update)
8598
8599            # Remove added columns
8600            for added_column in added_columns:
8601                self.drop_column(column=added_column)
8602
8603            # Delete dataframe
8604            del dataframe_genotypeconcordance
8605            gc.collect()

The function calculation_genotype_concordance calculates the genotype concordance for multi-caller VCF files and updates the variant information in the database.

def calculation_barcode(self, tag: str = 'barcode') -> None: View Source

8607    def calculation_barcode(self, tag: str = "barcode") -> None:
8608        """
8609        The `calculation_barcode` function calculates barcode values for variants in a VCF file and
8610        updates the INFO field in the file with the calculated barcode values.
8611
8612        :param tag: The `tag` parameter in the `calculation_barcode` function is used to specify the tag
8613        name that will be used for the barcode calculation in the VCF file. If no tag name is provided,
8614        the default tag name is set to "barcode", defaults to barcode
8615        :type tag: str (optional)
8616        """
8617
8618        # if FORMAT and samples
8619        if (
8620            "FORMAT" in self.get_header_columns_as_list()
8621            and self.get_header_sample_list()
8622        ):
8623
8624            # barcode annotation field
8625            if not tag:
8626                tag = "barcode"
8627
8628            # VCF infos tags
8629            vcf_infos_tags = {
8630                tag: "barcode calculation (VaRank)",
8631            }
8632
8633            # Prefix
8634            prefix = self.get_explode_infos_prefix()
8635
8636            # Field
8637            barcode_infos = prefix + tag
8638
8639            # Variants table
8640            table_variants = self.get_table_variants()
8641
8642            # Header
8643            vcf_reader = self.get_header()
8644
8645            # Create variant id
8646            variant_id_column = self.get_variant_id_column()
8647            added_columns = [variant_id_column]
8648
8649            # variant_id, FORMAT and samples
8650            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
8651                self.get_header_sample_list()
8652            )
8653
8654            # Create dataframe
8655            dataframe_barcode = self.get_query_to_df(
8656                f""" SELECT {samples_fields} FROM {table_variants} """
8657            )
8658
8659            # Create barcode column
8660            dataframe_barcode[barcode_infos] = dataframe_barcode.apply(
8661                lambda row: barcode(row, samples=self.get_header_sample_list()), axis=1
8662            )
8663
8664            # Add barcode to header
8665            vcf_reader.infos[tag] = vcf.parser._Info(
8666                tag,
8667                ".",
8668                "String",
8669                vcf_infos_tags.get(tag, vcf_infos_tags.get(tag)),
8670                "howard calculation",
8671                "0",
8672                self.code_type_map.get("String"),
8673            )
8674
8675            # Update
8676            sql_update = f"""
8677                UPDATE {table_variants}
8678                SET "INFO" = 
8679                    concat(
8680                        CASE
8681                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
8682                            THEN ''
8683                            ELSE concat("INFO", ';')
8684                        END,
8685                        CASE
8686                            WHEN dataframe_barcode."{barcode_infos}" NOT IN ('','.')
8687                            AND dataframe_barcode."{barcode_infos}" NOT NULL
8688                            THEN concat(
8689                                    '{tag}=',
8690                                    dataframe_barcode."{barcode_infos}"
8691                                )
8692                            ELSE ''
8693                        END
8694                    )
8695                FROM dataframe_barcode
8696                WHERE {table_variants}."{variant_id_column}" = dataframe_barcode."{variant_id_column}"
8697            """
8698            self.conn.execute(sql_update)
8699
8700            # Remove added columns
8701            for added_column in added_columns:
8702                self.drop_column(column=added_column)
8703
8704            # Delete dataframe
8705            del dataframe_barcode
8706            gc.collect()

The calculation_barcode function calculates barcode values for variants in a VCF file and updates the INFO field in the file with the calculated barcode values.

Parameters

tag: The tag parameter in the calculation_barcode function is used to specify the tag name that will be used for the barcode calculation in the VCF file. If no tag name is provided, the default tag name is set to "barcode", defaults to barcode

def calculation_barcode_family(self, tag: str = 'BCF') -> None: View Source

8708    def calculation_barcode_family(self, tag: str = "BCF") -> None:
8709        """
8710        The `calculation_barcode_family` function calculates barcode values for variants in a VCF file
8711        and updates the INFO field in the file with the calculated barcode values.
8712
8713        :param tag: The `tag` parameter in the `calculation_barcode_family` function is used to specify
8714        the barcode tag that will be added to the VCF file during the calculation process. If no value
8715        is provided for the `tag` parameter, the default value used is "BCF", defaults to BCF
8716        :type tag: str (optional)
8717        """
8718
8719        # if FORMAT and samples
8720        if (
8721            "FORMAT" in self.get_header_columns_as_list()
8722            and self.get_header_sample_list()
8723        ):
8724
8725            # barcode annotation field
8726            if not tag:
8727                tag = "BCF"
8728
8729            # VCF infos tags
8730            vcf_infos_tags = {
8731                tag: "barcode family calculation",
8732                f"{tag}S": "barcode family samples",
8733            }
8734
8735            # Param
8736            param = self.get_param()
8737            log.debug(f"param={param}")
8738
8739            # Prefix
8740            prefix = self.get_explode_infos_prefix()
8741
8742            # PED param
8743            ped = (
8744                param.get("calculation", {})
8745                .get("calculations", {})
8746                .get("BARCODEFAMILY", {})
8747                .get("family_pedigree", None)
8748            )
8749            log.debug(f"ped={ped}")
8750
8751            # Load PED
8752            if ped:
8753
8754                # Pedigree is a file
8755                if isinstance(ped, str) and os.path.exists(full_path(ped)):
8756                    log.debug("Pedigree is file")
8757                    with open(full_path(ped)) as ped:
8758                        ped = json.load(ped)
8759
8760                # Pedigree is a string
8761                elif isinstance(ped, str):
8762                    log.debug("Pedigree is str")
8763                    try:
8764                        ped = json.loads(ped)
8765                        log.debug("Pedigree is json str")
8766                    except ValueError as e:
8767                        ped_samples = ped.split(",")
8768                        ped = {}
8769                        for ped_sample in ped_samples:
8770                            ped[ped_sample] = ped_sample
8771
8772                # Pedigree is a dict
8773                elif isinstance(ped, dict):
8774                    log.debug("Pedigree is dict")
8775
8776                # Pedigree is not well formatted
8777                else:
8778                    msg_error = "Pedigree not well formatted"
8779                    log.error(msg_error)
8780                    raise ValueError(msg_error)
8781
8782                # Construct list
8783                ped_samples = list(ped.values())
8784
8785            else:
8786                log.debug("Pedigree not defined. Take all samples")
8787                ped_samples = self.get_header_sample_list()
8788                ped = {}
8789                for ped_sample in ped_samples:
8790                    ped[ped_sample] = ped_sample
8791
8792            # Check pedigree
8793            if not ped or len(ped) == 0:
8794                msg_error = f"Error in pedigree: samples {ped_samples}"
8795                log.error(msg_error)
8796                raise ValueError(msg_error)
8797
8798            # Log
8799            log.info(
8800                "Calculation 'BARCODEFAMILY' - Samples: "
8801                + ", ".join([f"{member}='{ped[member]}'" for member in ped])
8802            )
8803            log.debug(f"ped_samples={ped_samples}")
8804
8805            # Field
8806            barcode_infos = prefix + tag
8807
8808            # Variants table
8809            table_variants = self.get_table_variants()
8810
8811            # Header
8812            vcf_reader = self.get_header()
8813
8814            # Create variant id
8815            variant_id_column = self.get_variant_id_column()
8816            added_columns = [variant_id_column]
8817
8818            # variant_id, FORMAT and samples
8819            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
8820                ped_samples
8821            )
8822
8823            # Create dataframe
8824            dataframe_barcode = self.get_query_to_df(
8825                f""" SELECT {samples_fields} FROM {table_variants} """
8826            )
8827
8828            # Create barcode column
8829            dataframe_barcode[barcode_infos] = dataframe_barcode.apply(
8830                lambda row: barcode(row, samples=ped_samples), axis=1
8831            )
8832
8833            # Add barcode family to header
8834            # Add vaf_normalization to header
8835            vcf_reader.formats[tag] = vcf.parser._Format(
8836                id=tag,
8837                num=".",
8838                type="String",
8839                desc=vcf_infos_tags.get(tag, "barcode family calculation"),
8840                type_code=self.code_type_map.get("String"),
8841            )
8842            vcf_reader.formats[f"{tag}S"] = vcf.parser._Format(
8843                id=f"{tag}S",
8844                num=".",
8845                type="String",
8846                desc=vcf_infos_tags.get(f"{tag}S", "barcode family samples"),
8847                type_code=self.code_type_map.get("String"),
8848            )
8849
8850            # Update
8851            # for sample in ped_samples:
8852            sql_update_set = []
8853            for sample in self.get_header_sample_list() + ["FORMAT"]:
8854                if sample in ped_samples:
8855                    value = f'dataframe_barcode."{barcode_infos}"'
8856                    value_samples = "'" + ",".join(ped_samples) + "'"
8857                elif sample == "FORMAT":
8858                    value = f"'{tag}'"
8859                    value_samples = f"'{tag}S'"
8860                else:
8861                    value = "'.'"
8862                    value_samples = "'.'"
8863                format_regex = r"[a-zA-Z0-9\s]"
8864                sql_update_set.append(
8865                    f"""
8866                        "{sample}" = 
8867                        concat(
8868                            CASE
8869                                WHEN {table_variants}."{sample}" = './.'
8870                                THEN concat('./.',regexp_replace(regexp_replace({table_variants}.FORMAT, '{format_regex}', '', 'g'), ':', ':.', 'g'))
8871                                ELSE {table_variants}."{sample}"
8872                            END,
8873                            ':',
8874                            {value},
8875                            ':',
8876                            {value_samples}
8877                        )
8878                    """
8879                )
8880
8881            sql_update_set_join = ", ".join(sql_update_set)
8882            sql_update = f"""
8883                UPDATE {table_variants}
8884                SET {sql_update_set_join}
8885                FROM dataframe_barcode
8886                WHERE {table_variants}."{variant_id_column}" = dataframe_barcode."{variant_id_column}"
8887            """
8888            self.conn.execute(sql_update)
8889
8890            # Remove added columns
8891            for added_column in added_columns:
8892                self.drop_column(column=added_column)
8893
8894            # Delete dataframe
8895            del dataframe_barcode
8896            gc.collect()

The calculation_barcode_family function calculates barcode values for variants in a VCF file and updates the INFO field in the file with the calculated barcode values.

Parameters

tag: The tag parameter in the calculation_barcode_family function is used to specify the barcode tag that will be added to the VCF file during the calculation process. If no value is provided for the tag parameter, the default value used is "BCF", defaults to BCF

def calculation_trio(self) -> None: View Source

8898    def calculation_trio(self) -> None:
8899        """
8900        The `calculation_trio` function performs trio calculations on a VCF file by adding trio
8901        information to the INFO field of each variant.
8902        """
8903
8904        # if FORMAT and samples
8905        if (
8906            "FORMAT" in self.get_header_columns_as_list()
8907            and self.get_header_sample_list()
8908        ):
8909
8910            # trio annotation field
8911            trio_tag = "trio"
8912
8913            # VCF infos tags
8914            vcf_infos_tags = {
8915                "trio": "trio calculation",
8916            }
8917
8918            # Param
8919            param = self.get_param()
8920
8921            # Prefix
8922            prefix = self.get_explode_infos_prefix()
8923
8924            # Trio param
8925            trio_ped = (
8926                param.get("calculation", {})
8927                .get("calculations", {})
8928                .get("TRIO", {})
8929                .get("trio_pedigree", None)
8930            )
8931
8932            # Load trio
8933            if trio_ped:
8934
8935                # Trio pedigree is a file
8936                if isinstance(trio_ped, str) and os.path.exists(full_path(trio_ped)):
8937                    log.debug("TRIO pedigree is file")
8938                    with open(full_path(trio_ped)) as trio_ped:
8939                        trio_ped = json.load(trio_ped)
8940
8941                # Trio pedigree is a string
8942                elif isinstance(trio_ped, str):
8943                    log.debug("TRIO pedigree is str")
8944                    try:
8945                        trio_ped = json.loads(trio_ped)
8946                        log.debug("TRIO pedigree is json str")
8947                    except ValueError as e:
8948                        trio_samples = trio_ped.split(",")
8949                        if len(trio_samples) == 3:
8950                            trio_ped = {
8951                                "father": trio_samples[0],
8952                                "mother": trio_samples[1],
8953                                "child": trio_samples[2],
8954                            }
8955                            log.debug("TRIO pedigree is list str")
8956                        else:
8957                            msg_error = "TRIO pedigree not well formatted"
8958                            log.error(msg_error)
8959                            raise ValueError(msg_error)
8960
8961                # Trio pedigree is a dict
8962                elif isinstance(trio_ped, dict):
8963                    log.debug("TRIO pedigree is dict")
8964
8965                # Trio pedigree is not well formatted
8966                else:
8967                    msg_error = "TRIO pedigree not well formatted"
8968                    log.error(msg_error)
8969                    raise ValueError(msg_error)
8970
8971                # Construct trio list
8972                trio_samples = [
8973                    trio_ped.get("father", ""),
8974                    trio_ped.get("mother", ""),
8975                    trio_ped.get("child", ""),
8976                ]
8977
8978            else:
8979                log.debug("TRIO pedigree not defined. Take the first 3 samples")
8980                samples_list = self.get_header_sample_list()
8981                if len(samples_list) >= 3:
8982                    trio_samples = self.get_header_sample_list()[0:3]
8983                    trio_ped = {
8984                        "father": trio_samples[0],
8985                        "mother": trio_samples[1],
8986                        "child": trio_samples[2],
8987                    }
8988                else:
8989                    msg_error = f"Error in TRIO pedigree: only {len(samples_list)} samples {samples_list}"
8990                    log.error(msg_error)
8991                    raise ValueError(msg_error)
8992
8993            # Check trio pedigree
8994            if not trio_ped or len(trio_ped) != 3:
8995                msg_error = f"Error in TRIO pedigree: {trio_ped}"
8996                log.error(msg_error)
8997                raise ValueError(msg_error)
8998
8999            # Log
9000            log.info(
9001                f"Calculation 'TRIO' - Samples: "
9002                + ", ".join([f"{member}='{trio_ped[member]}'" for member in trio_ped])
9003            )
9004
9005            # Field
9006            trio_infos = prefix + trio_tag
9007
9008            # Variants table
9009            table_variants = self.get_table_variants()
9010
9011            # Header
9012            vcf_reader = self.get_header()
9013
9014            # Create variant id
9015            variant_id_column = self.get_variant_id_column()
9016            added_columns = [variant_id_column]
9017
9018            # variant_id, FORMAT and samples
9019            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9020                self.get_header_sample_list()
9021            )
9022
9023            # Create dataframe
9024            dataframe_trio = self.get_query_to_df(
9025                f""" SELECT {samples_fields} FROM {table_variants} """
9026            )
9027
9028            # Create trio column
9029            dataframe_trio[trio_infos] = dataframe_trio.apply(
9030                lambda row: trio(row, samples=trio_samples), axis=1
9031            )
9032
9033            # Add trio to header
9034            vcf_reader.infos[trio_tag] = vcf.parser._Info(
9035                trio_tag,
9036                ".",
9037                "String",
9038                vcf_infos_tags.get(trio_tag, "snpEff hgvs annotations"),
9039                "howard calculation",
9040                "0",
9041                self.code_type_map.get("String"),
9042            )
9043
9044            # Update
9045            sql_update = f"""
9046                UPDATE {table_variants}
9047                SET "INFO" = 
9048                    concat(
9049                        CASE
9050                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9051                            THEN ''
9052                            ELSE concat("INFO", ';')
9053                        END,
9054                        CASE
9055                            WHEN dataframe_trio."{trio_infos}" NOT IN ('','.')
9056                             AND dataframe_trio."{trio_infos}" NOT NULL
9057                            THEN concat(
9058                                    '{trio_tag}=',
9059                                    dataframe_trio."{trio_infos}"
9060                                )
9061                            ELSE ''
9062                        END
9063                    )
9064                FROM dataframe_trio
9065                WHERE {table_variants}."{variant_id_column}" = dataframe_trio."{variant_id_column}"
9066            """
9067            self.conn.execute(sql_update)
9068
9069            # Remove added columns
9070            for added_column in added_columns:
9071                self.drop_column(column=added_column)
9072
9073            # Delete dataframe
9074            del dataframe_trio
9075            gc.collect()

The calculation_trio function performs trio calculations on a VCF file by adding trio information to the INFO field of each variant.

def calculation_vaf_normalization(self) -> None: View Source

9077    def calculation_vaf_normalization(self) -> None:
9078        """
9079        The `calculation_vaf_normalization` function calculates the VAF (Variant Allele Frequency)
9080        normalization for each sample in a VCF file and updates the FORMAT and INFO fields accordingly.
9081        :return: The function does not return anything.
9082        """
9083
9084        # if FORMAT and samples
9085        if (
9086            "FORMAT" in self.get_header_columns_as_list()
9087            and self.get_header_sample_list()
9088        ):
9089
9090            # vaf_normalization annotation field
9091            vaf_normalization_tag = "VAF"
9092
9093            # VCF infos tags
9094            vcf_infos_tags = {
9095                "VAF": "VAF Variant Frequency",
9096            }
9097
9098            # Prefix
9099            prefix = self.get_explode_infos_prefix()
9100
9101            # Variants table
9102            table_variants = self.get_table_variants()
9103
9104            # Header
9105            vcf_reader = self.get_header()
9106
9107            # Do not calculate if VAF already exists
9108            if "VAF" in vcf_reader.formats:
9109                log.debug("VAF already on genotypes")
9110                return
9111
9112            # Create variant id
9113            variant_id_column = self.get_variant_id_column()
9114            added_columns = [variant_id_column]
9115
9116            # variant_id, FORMAT and samples
9117            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9118                f""" "{sample}" """ for sample in self.get_header_sample_list()
9119            )
9120
9121            # Create dataframe
9122            query = f""" SELECT {variant_id_column}, FORMAT, {samples_fields} FROM {table_variants} """
9123            log.debug(f"query={query}")
9124            dataframe_vaf_normalization = self.get_query_to_df(query=query)
9125
9126            vaf_normalization_set = []
9127
9128            # for each sample vaf_normalization
9129            for sample in self.get_header_sample_list():
9130                dataframe_vaf_normalization[sample] = dataframe_vaf_normalization.apply(
9131                    lambda row: vaf_normalization(row, sample=sample), axis=1
9132                )
9133                vaf_normalization_set.append(
9134                    f""" "{sample}" = dataframe_vaf_normalization."{sample}" """
9135                )
9136
9137            # Add VAF to FORMAT
9138            dataframe_vaf_normalization["FORMAT"] = dataframe_vaf_normalization[
9139                "FORMAT"
9140            ].apply(lambda x: str(x) + ":VAF")
9141            vaf_normalization_set.append(
9142                f""" "FORMAT" = dataframe_vaf_normalization."FORMAT" """
9143            )
9144
9145            # Add vaf_normalization to header
9146            vcf_reader.formats[vaf_normalization_tag] = vcf.parser._Format(
9147                id=vaf_normalization_tag,
9148                num="1",
9149                type="Float",
9150                desc=vcf_infos_tags.get(vaf_normalization_tag, "VAF Variant Frequency"),
9151                type_code=self.code_type_map.get("Float"),
9152            )
9153
9154            # Create fields to add in INFO
9155            sql_vaf_normalization_set = " , ".join(vaf_normalization_set)
9156
9157            # Update
9158            sql_update = f"""
9159                UPDATE {table_variants}
9160                SET {sql_vaf_normalization_set}
9161                FROM dataframe_vaf_normalization
9162                WHERE variants."{variant_id_column}" = dataframe_vaf_normalization."{variant_id_column}"
9163
9164            """
9165            self.conn.execute(sql_update)
9166
9167            # Remove added columns
9168            for added_column in added_columns:
9169                self.drop_column(column=added_column)
9170
9171            # Delete dataframe
9172            del dataframe_vaf_normalization
9173            gc.collect()

The calculation_vaf_normalization function calculates the VAF (Variant Allele Frequency) normalization for each sample in a VCF file and updates the FORMAT and INFO fields accordingly.

Returns

The function does not return anything.

def calculation_genotype_stats(self, info: str = 'VAF') -> None: View Source

9175    def calculation_genotype_stats(self, info: str = "VAF") -> None:
9176        """
9177        The `calculation_genotype_stats` function calculates genotype statistics for a given information
9178        field in a VCF file and updates the INFO column of the variants table with the calculated
9179        statistics.
9180
9181        :param info: The `info` parameter is a string that represents the type of information for which
9182        genotype statistics are calculated. It is used to generate various VCF info tags for the
9183        statistics, such as the number of occurrences, the list of values, the minimum value, the
9184        maximum value, the mean, the median, defaults to VAF
9185        :type info: str (optional)
9186        """
9187
9188        # if FORMAT and samples
9189        if (
9190            "FORMAT" in self.get_header_columns_as_list()
9191            and self.get_header_sample_list()
9192        ):
9193
9194            # vaf_stats annotation field
9195            vaf_stats_tag = info + "_stats"
9196
9197            # VCF infos tags
9198            vcf_infos_tags = {
9199                info + "_stats_nb": f"genotype {info} Statistics - number of {info}",
9200                info + "_stats_list": f"genotype {info} Statistics - list of {info}",
9201                info + "_stats_min": f"genotype {info} Statistics - min {info}",
9202                info + "_stats_max": f"genotype {info} Statistics - max {info}",
9203                info + "_stats_mean": f"genotype {info} Statistics - mean {info}",
9204                info + "_stats_mediane": f"genotype {info} Statistics - mediane {info}",
9205                info
9206                + "_stats_stdev": f"genotype {info} Statistics - standard deviation {info}",
9207            }
9208
9209            # Prefix
9210            prefix = self.get_explode_infos_prefix()
9211
9212            # Field
9213            vaf_stats_infos = prefix + vaf_stats_tag
9214
9215            # Variants table
9216            table_variants = self.get_table_variants()
9217
9218            # Header
9219            vcf_reader = self.get_header()
9220
9221            # Create variant id
9222            variant_id_column = self.get_variant_id_column()
9223            added_columns = [variant_id_column]
9224
9225            # variant_id, FORMAT and samples
9226            samples_fields = f" {variant_id_column}, FORMAT , " + " , ".join(
9227                self.get_header_sample_list()
9228            )
9229
9230            # Create dataframe
9231            dataframe_vaf_stats = self.get_query_to_df(
9232                f""" SELECT {samples_fields} FROM {table_variants} """
9233            )
9234
9235            # Create vaf_stats column
9236            dataframe_vaf_stats[vaf_stats_infos] = dataframe_vaf_stats.apply(
9237                lambda row: genotype_stats(
9238                    row, samples=self.get_header_sample_list(), info=info
9239                ),
9240                axis=1,
9241            )
9242
9243            # List of vcf tags
9244            sql_vaf_stats_fields = []
9245
9246            # Check all VAF stats infos
9247            for stat in vcf_infos_tags:
9248
9249                # Extract stats
9250                dataframe_vaf_stats[stat] = dataframe_vaf_stats[vaf_stats_infos].apply(
9251                    lambda x: dict(x).get(stat, "")
9252                )
9253
9254                # Add snpeff_hgvs to header
9255                vcf_reader.infos[stat] = vcf.parser._Info(
9256                    stat,
9257                    ".",
9258                    "String",
9259                    vcf_infos_tags.get(stat, "genotype statistics"),
9260                    "howard calculation",
9261                    "0",
9262                    self.code_type_map.get("String"),
9263                )
9264
9265                if len(sql_vaf_stats_fields):
9266                    sep = ";"
9267                else:
9268                    sep = ""
9269
9270                # Create fields to add in INFO
9271                sql_vaf_stats_fields.append(
9272                    f"""
9273                        CASE
9274                            WHEN dataframe_vaf_stats."{stat}" NOT NULL
9275                            THEN concat(
9276                                    '{sep}{stat}=',
9277                                    dataframe_vaf_stats."{stat}"
9278                                )
9279                            ELSE ''
9280                        END
9281                    """
9282                )
9283
9284            # SQL set for update
9285            sql_vaf_stats_fields_set = ",  ".join(sql_vaf_stats_fields)
9286
9287            # Update
9288            sql_update = f"""
9289                UPDATE {table_variants}
9290                SET "INFO" = 
9291                    concat(
9292                        CASE
9293                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9294                            THEN ''
9295                            ELSE concat("INFO", ';')
9296                        END,
9297                        {sql_vaf_stats_fields_set}
9298                    )
9299                FROM dataframe_vaf_stats
9300                WHERE {table_variants}."{variant_id_column}" = dataframe_vaf_stats."{variant_id_column}"
9301
9302            """
9303            self.conn.execute(sql_update)
9304
9305            # Remove added columns
9306            for added_column in added_columns:
9307                self.drop_column(column=added_column)
9308
9309            # Delete dataframe
9310            del dataframe_vaf_stats
9311            gc.collect()

The calculation_genotype_stats function calculates genotype statistics for a given information field in a VCF file and updates the INFO column of the variants table with the calculated statistics.

Parameters

info: The info parameter is a string that represents the type of information for which genotype statistics are calculated. It is used to generate various VCF info tags for the statistics, such as the number of occurrences, the list of values, the minimum value, the maximum value, the mean, the median, defaults to VAF

def calculation_transcripts_annotation(self, info_json: str = None, info_format: str = None) -> None: View Source

9313    def calculation_transcripts_annotation(
9314        self, info_json: str = None, info_format: str = None
9315    ) -> None:
9316        """
9317        The `calculation_transcripts_annotation` function creates a transcripts table and adds an info
9318        field to it if transcripts are available.
9319
9320        :param info_json: The `info_json` parameter in the `calculation_transcripts_annotation` method
9321        is a string parameter that represents the information field to be used in the transcripts JSON.
9322        It is used to specify the JSON format for the transcripts information. If no value is provided
9323        when calling the method, it defaults to "
9324        :type info_json: str
9325        :param info_format: The `info_format` parameter in the `calculation_transcripts_annotation`
9326        method is a string parameter that specifies the format of the information field to be used in
9327        the transcripts JSON. It is used to define the format of the information field
9328        :type info_format: str
9329        """
9330
9331        # Create transcripts table
9332        transcripts_table = self.create_transcript_view()
9333
9334        # Add info field
9335        if transcripts_table:
9336            self.transcript_view_to_variants(
9337                transcripts_table=transcripts_table,
9338                transcripts_info_field_json=info_json,
9339                transcripts_info_field_format=info_format,
9340            )
9341        else:
9342            log.info("No Transcripts to process. Check param.json file configuration")

The calculation_transcripts_annotation function creates a transcripts table and adds an info field to it if transcripts are available.

Parameters

info_json: The info_json parameter in the calculation_transcripts_annotation method is a string parameter that represents the information field to be used in the transcripts JSON. It is used to specify the JSON format for the transcripts information. If no value is provided when calling the method, it defaults to "
info_format: The info_format parameter in the calculation_transcripts_annotation method is a string parameter that specifies the format of the information field to be used in the transcripts JSON. It is used to define the format of the information field

def calculation_transcripts_prioritization(self) -> None: View Source

9344    def calculation_transcripts_prioritization(self) -> None:
9345        """
9346        The function `calculation_transcripts_prioritization` creates a transcripts table and
9347        prioritizes transcripts based on certain criteria.
9348        """
9349
9350        # Create transcripts table
9351        transcripts_table = self.create_transcript_view()
9352
9353        # Add info field
9354        if transcripts_table:
9355            self.transcripts_prioritization(transcripts_table=transcripts_table)
9356        else:
9357            log.info("No Transcripts to process. Check param.json file configuration")

The function calculation_transcripts_prioritization creates a transcripts table and prioritizes transcripts based on certain criteria.

def transcripts_prioritization(self, transcripts_table: str = None, param: dict = {}) -> bool: View Source

9363    def transcripts_prioritization(
9364        self, transcripts_table: str = None, param: dict = {}
9365    ) -> bool:
9366        """
9367        The `transcripts_prioritization` function prioritizes transcripts based on certain parameters
9368        and updates the variants table with the prioritized information.
9369
9370        :param transcripts_table: The `transcripts_table` parameter is a string that specifies the name
9371        of the table containing transcripts data. If no value is provided, it defaults to "transcripts".
9372        This parameter is used to identify the table where the transcripts data is stored for the
9373        prioritization process
9374        :type transcripts_table: str
9375        :param param: The `param` parameter in the `transcripts_prioritization` method is a dictionary
9376        that contains various configuration settings for the prioritization process of transcripts. It
9377        is used to customize the behavior of the prioritization algorithm and includes settings such as
9378        the prefix for prioritization fields, default profiles, and other
9379        :type param: dict
9380        :return: The function `transcripts_prioritization` returns a boolean value `True` if the
9381        transcripts prioritization process is successfully completed, and `False` if there are any
9382        issues or if no profile is defined for transcripts prioritization.
9383        """
9384
9385        log.debug("Start transcripts prioritization...")
9386
9387        # Param
9388        if not param:
9389            param = self.get_param()
9390
9391        # Variants table
9392        table_variants = self.get_table_variants()
9393        log.debug(f"transcripts_table={transcripts_table}")
9394        # Transcripts table
9395        if transcripts_table is None:
9396            log.debug(f"transcripts_table={transcripts_table}")
9397            transcripts_table = self.create_transcript_view(
9398                transcripts_table="transcripts", param=param
9399            )
9400            log.debug(f"transcripts_table={transcripts_table}")
9401        if transcripts_table is None:
9402            msg_err = "No Transcripts table availalble"
9403            log.error(msg_err)
9404            raise ValueError(msg_err)
9405
9406        # Get transcripts columns
9407        columns_as_list_query = f"""
9408            DESCRIBE {transcripts_table}
9409        """
9410        columns_as_list = list(
9411            self.get_query_to_df(columns_as_list_query)["column_name"]
9412        )
9413
9414        # Create INFO if not exists
9415        if "INFO" not in columns_as_list:
9416            query_add_info = f"""
9417                ALTER TABLE {transcripts_table} ADD COLUMN INFO STRING DEFAULT '';
9418            """
9419            self.execute_query(query_add_info)
9420
9421        # Prioritization param and Force only PZ Score and Flag
9422        pz_param = param.get("transcripts", {}).get("prioritization", {})
9423        pz_fields_score = pz_param.get("pzprefix", "PTZ") + "Score"
9424        pz_fields_flag = pz_param.get("pzprefix", "PTZ") + "Flag"
9425        pz_fields_transcripts = pz_param.get("pzprefix", "PTZ") + "Transcript"
9426        pz_param["pzfields"] = [pz_fields_score, pz_fields_flag]
9427        pz_profile_default = (
9428            param.get("transcripts", {}).get("prioritization", {}).get("profiles", None)
9429        )
9430
9431        # Exit if no profile
9432        if pz_profile_default is None:
9433            log.warning("No profile defined for transcripts prioritization")
9434            return False
9435
9436        # Prioritization
9437        prioritization_result = self.prioritization(
9438            table=transcripts_table,
9439            pz_param=param.get("transcripts", {}).get("prioritization", {}),
9440        )
9441        if not prioritization_result:
9442            log.warning("Transcripts prioritization not processed")
9443            return False
9444
9445        # Explode PZ fields
9446        self.explode_infos(
9447            table=transcripts_table,
9448            fields=param.get("transcripts", {})
9449            .get("prioritization", {})
9450            .get("pzfields", []),
9451        )
9452
9453        # Export Transcripts prioritization infos to variants table
9454        query_update = f"""
9455            WITH RankedTranscripts AS (
9456                SELECT
9457                    "#CHROM", POS, REF, ALT, transcript, {pz_fields_score}, {pz_fields_flag},
9458                    ROW_NUMBER() OVER (
9459                        PARTITION BY "#CHROM", POS, REF, ALT
9460                        ORDER BY {pz_fields_flag} ASC, {pz_fields_score} DESC, transcript ASC
9461                    ) AS rn
9462                FROM
9463                    {transcripts_table}
9464            )
9465            UPDATE {table_variants}
9466                SET
9467                INFO = CONCAT(CASE
9468                            WHEN "INFO" IS NULL OR "INFO" IN ('','.')
9469                            THEN ''
9470                            ELSE concat("INFO", ';')
9471                        END,
9472                        concat('{pz_fields_transcripts}=', transcript, ';{pz_fields_score}=', {pz_fields_score}, ';{pz_fields_flag}=', {pz_fields_flag})
9473                        )
9474            FROM
9475                RankedTranscripts
9476            WHERE
9477                rn = 1
9478                AND variants."#CHROM" = RankedTranscripts."#CHROM"
9479                AND variants."POS" = RankedTranscripts."POS"
9480                AND variants."REF" = RankedTranscripts."REF"
9481                AND variants."ALT" = RankedTranscripts."ALT"
9482                
9483        """
9484        self.execute_query(query=query_update)
9485
9486        # Add PZ Transcript in header
9487        self.get_header().infos[pz_fields_transcripts] = vcf.parser._Info(
9488            pz_fields_transcripts,
9489            ".",
9490            "String",
9491            f"Transcript selected from transcripts prioritization process, profile {pz_profile_default}",
9492            "unknown",
9493            "unknown",
9494            code_type_map["String"],
9495        )
9496
9497        # Return
9498        return True

The transcripts_prioritization function prioritizes transcripts based on certain parameters and updates the variants table with the prioritized information.

Parameters

transcripts_table: The transcripts_table parameter is a string that specifies the name of the table containing transcripts data. If no value is provided, it defaults to "transcripts". This parameter is used to identify the table where the transcripts data is stored for the prioritization process
param: The param parameter in the transcripts_prioritization method is a dictionary that contains various configuration settings for the prioritization process of transcripts. It is used to customize the behavior of the prioritization algorithm and includes settings such as the prefix for prioritization fields, default profiles, and other

Returns

The function transcripts_prioritization returns a boolean value True if the transcripts prioritization process is successfully completed, and False if there are any issues or if no profile is defined for transcripts prioritization.

def create_transcript_view_from_columns_map( self, transcripts_table: str = 'transcripts', columns_maps: dict = {}, added_columns: list = [], temporary_tables: list = None, annotation_fields: list = None) -> tuple[list, list, list]: View Source

9500    def create_transcript_view_from_columns_map(
9501        self,
9502        transcripts_table: str = "transcripts",
9503        columns_maps: dict = {},
9504        added_columns: list = [],
9505        temporary_tables: list = None,
9506        annotation_fields: list = None,
9507    ) -> tuple[list, list, list]:
9508        """
9509        The `create_transcript_view_from_columns_map` function generates a temporary table view based on
9510        specified columns mapping for transcripts data.
9511
9512        :param transcripts_table: The `transcripts_table` parameter is a string that specifies the name of
9513        the table where the transcripts data is stored or will be stored in the database. This table
9514        typically contains information about transcripts such as Ensembl transcript IDs, gene names, scores,
9515        predictions, etc. It defaults to "transcripts, defaults to transcripts
9516        :type transcripts_table: str (optional)
9517        :param columns_maps: The `columns_maps` parameter is a dictionary that contains information about
9518        how to map columns from a transcripts table to create a view. Each entry in the `columns_maps` list
9519        represents a mapping configuration for a specific set of columns. It typically includes details such
9520        as the main transcript column and additional information columns
9521        :type columns_maps: dict
9522        :param added_columns: The `added_columns` parameter in the `create_transcript_view_from_columns_map`
9523        function is a list that stores the additional columns that will be added to the view being created
9524        based on the columns map provided. These columns are generated by exploding the transcript
9525        information columns along with the main transcript column
9526        :type added_columns: list
9527        :param temporary_tables: The `temporary_tables` parameter in the
9528        `create_transcript_view_from_columns_map` function is a list that stores the names of temporary
9529        tables created during the process of creating a transcript view from a columns map. These temporary
9530        tables are used to store intermediate results or transformations before the final view is generated
9531        :type temporary_tables: list
9532        :param annotation_fields: The `annotation_fields` parameter in the
9533        `create_transcript_view_from_columns_map` function is a list that stores the fields that are used
9534        for annotation in the query view creation process. These fields are extracted from the
9535        `transcripts_column` and `transcripts_infos_columns` specified in the `columns
9536        :type annotation_fields: list
9537        :return: The function `create_transcript_view_from_columns_map` returns a tuple containing three
9538        lists: `added_columns`, `temporary_tables`, and `annotation_fields`.
9539        """
9540
9541        log.debug("Start transcrpts view creation from columns map...")
9542
9543        # "from_columns_map": [
9544        #     {
9545        #         "transcripts_column": "Ensembl_transcriptid",
9546        #         "transcripts_infos_columns": [
9547        #             "genename",
9548        #             "Ensembl_geneid",
9549        #             "LIST_S2_score",
9550        #             "LIST_S2_pred",
9551        #         ],
9552        #     },
9553        #     {
9554        #         "transcripts_column": "Ensembl_transcriptid",
9555        #         "transcripts_infos_columns": [
9556        #             "genename",
9557        #             "VARITY_R_score",
9558        #             "Aloft_pred",
9559        #         ],
9560        #     },
9561        # ],
9562
9563        # Init
9564        if temporary_tables is None:
9565            temporary_tables = []
9566        if annotation_fields is None:
9567            annotation_fields = []
9568
9569        # Variants table
9570        table_variants = self.get_table_variants()
9571
9572        for columns_map in columns_maps:
9573
9574            # Transcript column
9575            transcripts_column = columns_map.get("transcripts_column", None)
9576
9577            # Transcripts infos columns
9578            transcripts_infos_columns = columns_map.get("transcripts_infos_columns", [])
9579
9580            if transcripts_column is not None:
9581
9582                # Explode
9583                added_columns += self.explode_infos(
9584                    fields=[transcripts_column] + transcripts_infos_columns
9585                )
9586
9587                # View clauses
9588                clause_select = []
9589                for field in [transcripts_column] + transcripts_infos_columns:
9590                    clause_select.append(
9591                        f""" regexp_split_to_table("{field}", ',') AS '{field}' """
9592                    )
9593                    if field not in [transcripts_column]:
9594                        annotation_fields.append(field)
9595
9596                # Querey View
9597                query = f""" 
9598                    SELECT
9599                        "#CHROM", POS, REF, ALT,
9600                        "{transcripts_column}" AS 'transcript',
9601                        {", ".join(clause_select)}
9602                    FROM (
9603                        SELECT 
9604                            "#CHROM", POS, REF, ALT,
9605                            {", ".join(clause_select)}
9606                        FROM {table_variants}
9607                        )
9608                    WHERE "{transcripts_column}" IS NOT NULL
9609                """
9610
9611                # Create temporary table
9612                temporary_table = transcripts_table + "".join(
9613                    random.choices(string.ascii_uppercase + string.digits, k=10)
9614                )
9615
9616                # Temporary_tables
9617                temporary_tables.append(temporary_table)
9618                query_view = f"""
9619                    CREATE TEMPORARY TABLE {temporary_table}
9620                    AS ({query})
9621                """
9622                self.execute_query(query=query_view)
9623
9624        return added_columns, temporary_tables, annotation_fields

The create_transcript_view_from_columns_map function generates a temporary table view based on specified columns mapping for transcripts data.

Parameters

transcripts_table: The transcripts_table parameter is a string that specifies the name of the table where the transcripts data is stored or will be stored in the database. This table typically contains information about transcripts such as Ensembl transcript IDs, gene names, scores, predictions, etc. It defaults to "transcripts, defaults to transcripts
columns_maps: The columns_maps parameter is a dictionary that contains information about how to map columns from a transcripts table to create a view. Each entry in the columns_maps list represents a mapping configuration for a specific set of columns. It typically includes details such as the main transcript column and additional information columns
added_columns: The added_columns parameter in the create_transcript_view_from_columns_map function is a list that stores the additional columns that will be added to the view being created based on the columns map provided. These columns are generated by exploding the transcript information columns along with the main transcript column
temporary_tables: The temporary_tables parameter in the create_transcript_view_from_columns_map function is a list that stores the names of temporary tables created during the process of creating a transcript view from a columns map. These temporary tables are used to store intermediate results or transformations before the final view is generated
annotation_fields: The annotation_fields parameter in the create_transcript_view_from_columns_map function is a list that stores the fields that are used for annotation in the query view creation process. These fields are extracted from the transcripts_column and transcripts_infos_columns specified in the `columns

Returns

The function create_transcript_view_from_columns_map returns a tuple containing three lists: added_columns, temporary_tables, and annotation_fields.

def create_transcript_view_from_column_format( self, transcripts_table: str = 'transcripts', column_formats: dict = {}, temporary_tables: list = None, annotation_fields: list = None) -> tuple[list, list, list]: View Source

9626    def create_transcript_view_from_column_format(
9627        self,
9628        transcripts_table: str = "transcripts",
9629        column_formats: dict = {},
9630        temporary_tables: list = None,
9631        annotation_fields: list = None,
9632    ) -> tuple[list, list, list]:
9633        """
9634        The `create_transcript_view_from_column_format` function generates a transcript view based on
9635        specified column formats, adds additional columns and annotation fields, and returns the list of
9636        temporary tables and annotation fields.
9637
9638        :param transcripts_table: The `transcripts_table` parameter is a string that specifies the name of
9639        the table containing the transcripts data. This table will be used as the base table for creating
9640        the transcript view. The default value for this parameter is "transcripts", but you can provide a
9641        different table name if needed, defaults to transcripts
9642        :type transcripts_table: str (optional)
9643        :param column_formats: The `column_formats` parameter is a dictionary that contains information
9644        about the columns to be used for creating the transcript view. Each entry in the dictionary
9645        specifies the mapping between a transcripts column and a transcripts infos column. For example, in
9646        the provided code snippet:
9647        :type column_formats: dict
9648        :param temporary_tables: The `temporary_tables` parameter in the
9649        `create_transcript_view_from_column_format` function is a list that stores the names of temporary
9650        views created during the process of creating a transcript view from a column format. These temporary
9651        views are used to manipulate and extract data before generating the final transcript view. It
9652        :type temporary_tables: list
9653        :param annotation_fields: The `annotation_fields` parameter in the
9654        `create_transcript_view_from_column_format` function is a list that stores the annotation fields
9655        that are extracted from the temporary views created during the process. These annotation fields are
9656        obtained by querying the temporary views and extracting the column names excluding specific columns
9657        like `#CH
9658        :type annotation_fields: list
9659        :return: The `create_transcript_view_from_column_format` function returns two lists:
9660        `temporary_tables` and `annotation_fields`.
9661        """
9662
9663        log.debug("Start transcrpts view creation from column format...")
9664
9665        #  "from_column_format": [
9666        #     {
9667        #         "transcripts_column": "ANN",
9668        #         "transcripts_infos_column": "Feature_ID",
9669        #     }
9670        # ],
9671
9672        # Init
9673        if temporary_tables is None:
9674            temporary_tables = []
9675        if annotation_fields is None:
9676            annotation_fields = []
9677
9678        for column_format in column_formats:
9679
9680            # annotation field and transcript annotation field
9681            annotation_field = column_format.get("transcripts_column", "ANN")
9682            transcript_annotation = column_format.get(
9683                "transcripts_infos_column", "Feature_ID"
9684            )
9685
9686            # Temporary View name
9687            temporary_view_name = transcripts_table + "".join(
9688                random.choices(string.ascii_uppercase + string.digits, k=10)
9689            )
9690
9691            # Create temporary view name
9692            temporary_view_name = self.annotation_format_to_table(
9693                uniquify=True,
9694                annotation_field=annotation_field,
9695                view_name=temporary_view_name,
9696                annotation_id=transcript_annotation,
9697            )
9698
9699            # Annotation fields
9700            if temporary_view_name:
9701                query_annotation_fields = f"""
9702                    SELECT *
9703                    FROM (
9704                        DESCRIBE SELECT *
9705                        FROM {temporary_view_name}
9706                        )
9707                        WHERE column_name not in ('#CHROM', 'POS', 'REF', 'ALT')
9708                """
9709                df_annotation_fields = self.get_query_to_df(
9710                    query=query_annotation_fields
9711                )
9712
9713                # Add temporary view and annotation fields
9714                temporary_tables.append(temporary_view_name)
9715                annotation_fields += list(set(df_annotation_fields["column_name"]))
9716
9717        return temporary_tables, annotation_fields

The create_transcript_view_from_column_format function generates a transcript view based on specified column formats, adds additional columns and annotation fields, and returns the list of temporary tables and annotation fields.

Parameters

transcripts_table: The transcripts_table parameter is a string that specifies the name of the table containing the transcripts data. This table will be used as the base table for creating the transcript view. The default value for this parameter is "transcripts", but you can provide a different table name if needed, defaults to transcripts
column_formats: The column_formats parameter is a dictionary that contains information about the columns to be used for creating the transcript view. Each entry in the dictionary specifies the mapping between a transcripts column and a transcripts infos column. For example, in the provided code snippet:
temporary_tables: The temporary_tables parameter in the create_transcript_view_from_column_format function is a list that stores the names of temporary views created during the process of creating a transcript view from a column format. These temporary views are used to manipulate and extract data before generating the final transcript view. It
annotation_fields: The annotation_fields parameter in the create_transcript_view_from_column_format function is a list that stores the annotation fields that are extracted from the temporary views created during the process. These annotation fields are obtained by querying the temporary views and extracting the column names excluding specific columns like `#CH

Returns

The create_transcript_view_from_column_format function returns two lists: temporary_tables and annotation_fields.

def create_transcript_view( self, transcripts_table: str = None, transcripts_table_drop: bool = True, param: dict = {}) -> str: View Source

9719    def create_transcript_view(
9720        self,
9721        transcripts_table: str = None,
9722        transcripts_table_drop: bool = True,
9723        param: dict = {},
9724    ) -> str:
9725        """
9726        The `create_transcript_view` function generates a transcript view by processing data from a
9727        specified table based on provided parameters and structural information.
9728
9729        :param transcripts_table: The `transcripts_table` parameter in the `create_transcript_view` function
9730        is used to specify the name of the table that will store the final transcript view data. If a table
9731        name is not provided, the function will create a new table to store the transcript view data, and by
9732        default,, defaults to transcripts
9733        :type transcripts_table: str (optional)
9734        :param transcripts_table_drop: The `transcripts_table_drop` parameter in the
9735        `create_transcript_view` function is a boolean parameter that determines whether to drop the
9736        existing transcripts table before creating a new one. If `transcripts_table_drop` is set to `True`,
9737        the function will drop the existing transcripts table if it exists, defaults to True
9738        :type transcripts_table_drop: bool (optional)
9739        :param param: The `param` parameter in the `create_transcript_view` function is a dictionary that
9740        contains information needed to create a transcript view. It includes details such as the structure
9741        of the transcripts, columns mapping, column formats, and other necessary information for generating
9742        the view. This parameter allows for flexibility and customization
9743        :type param: dict
9744        :return: The `create_transcript_view` function returns the name of the transcripts table that was
9745        created or modified during the execution of the function.
9746        """
9747
9748        log.debug("Start transcripts view creation...")
9749
9750        # Default
9751        transcripts_table_default = "transcripts"
9752
9753        # Param
9754        if not param:
9755            param = self.get_param()
9756
9757        # Struct
9758        struct = param.get("transcripts", {}).get("struct", None)
9759
9760        if struct:
9761
9762            # Transcripts table
9763            if transcripts_table is None:
9764                transcripts_table = param.get("transcripts", {}).get(
9765                    "table", transcripts_table_default
9766                )
9767
9768            # added_columns
9769            added_columns = []
9770
9771            # Temporary tables
9772            temporary_tables = []
9773
9774            # Annotation fields
9775            annotation_fields = []
9776
9777            # from columns map
9778            columns_maps = struct.get("from_columns_map", [])
9779            added_columns_tmp, temporary_tables_tmp, annotation_fields_tmp = (
9780                self.create_transcript_view_from_columns_map(
9781                    transcripts_table=transcripts_table,
9782                    columns_maps=columns_maps,
9783                    added_columns=added_columns,
9784                    temporary_tables=temporary_tables,
9785                    annotation_fields=annotation_fields,
9786                )
9787            )
9788            added_columns += added_columns_tmp
9789            temporary_tables += temporary_tables_tmp
9790            annotation_fields += annotation_fields_tmp
9791
9792            # from column format
9793            column_formats = struct.get("from_column_format", [])
9794            temporary_tables_tmp, annotation_fields_tmp = (
9795                self.create_transcript_view_from_column_format(
9796                    transcripts_table=transcripts_table,
9797                    column_formats=column_formats,
9798                    temporary_tables=temporary_tables,
9799                    annotation_fields=annotation_fields,
9800                )
9801            )
9802            temporary_tables += temporary_tables_tmp
9803            annotation_fields += annotation_fields_tmp
9804
9805            # Merge temporary tables query
9806            query_merge = ""
9807            for temporary_table in temporary_tables:
9808
9809                # First temporary table
9810                if not query_merge:
9811                    query_merge = f"""
9812                        SELECT * FROM {temporary_table}
9813                    """
9814                # other temporary table (using UNION)
9815                else:
9816                    query_merge += f"""
9817                        UNION BY NAME SELECT * FROM {temporary_table}
9818                    """
9819
9820            # Merge on transcript
9821            query_merge_on_transcripts_annotation_fields = []
9822            # Aggregate all annotations fields
9823            for annotation_field in set(annotation_fields):
9824                query_merge_on_transcripts_annotation_fields.append(
9825                    f""" list_aggregate(list_distinct(array_agg({annotation_field})), 'string_agg', ',') AS {annotation_field} """
9826                )
9827            # Query for transcripts view
9828            query_merge_on_transcripts = f"""
9829                SELECT "#CHROM", POS, REF, ALT, transcript, {", ".join(query_merge_on_transcripts_annotation_fields)}
9830                FROM ({query_merge})
9831                GROUP BY "#CHROM", POS, REF, ALT, transcript
9832            """
9833
9834            # Drop transcript view is necessary
9835            if transcripts_table_drop:
9836                query_drop = f"""
9837                    DROP TABLE IF EXISTS {transcripts_table};
9838                """
9839                self.execute_query(query=query_drop)
9840
9841            # Merge and create transcript view
9842            query_create_view = f"""
9843                CREATE TABLE IF NOT EXISTS {transcripts_table}
9844                AS {query_merge_on_transcripts}
9845            """
9846            self.execute_query(query=query_create_view)
9847
9848            # Remove added columns
9849            for added_column in added_columns:
9850                self.drop_column(column=added_column)
9851
9852        else:
9853
9854            transcripts_table = None
9855
9856        return transcripts_table

The create_transcript_view function generates a transcript view by processing data from a specified table based on provided parameters and structural information.

Parameters

transcripts_table: The transcripts_table parameter in the create_transcript_view function is used to specify the name of the table that will store the final transcript view data. If a table name is not provided, the function will create a new table to store the transcript view data, and by default,, defaults to transcripts
transcripts_table_drop: The transcripts_table_drop parameter in the create_transcript_view function is a boolean parameter that determines whether to drop the existing transcripts table before creating a new one. If transcripts_table_drop is set to True, the function will drop the existing transcripts table if it exists, defaults to True
param: The param parameter in the create_transcript_view function is a dictionary that contains information needed to create a transcript view. It includes details such as the structure of the transcripts, columns mapping, column formats, and other necessary information for generating the view. This parameter allows for flexibility and customization

Returns

The create_transcript_view function returns the name of the transcripts table that was created or modified during the execution of the function.

def annotation_format_to_table( self, uniquify: bool = True, annotation_field: str = 'ANN', annotation_id: str = 'Feature_ID', view_name: str = 'transcripts') -> str: View Source

 9858    def annotation_format_to_table(
 9859        self,
 9860        uniquify: bool = True,
 9861        annotation_field: str = "ANN",
 9862        annotation_id: str = "Feature_ID",
 9863        view_name: str = "transcripts",
 9864    ) -> str:
 9865        """
 9866        The function `annotation_format_to_table` converts annotation data from a VCF file into a structured
 9867        table format.
 9868
 9869        :param uniquify: The `uniquify` parameter is a boolean flag that determines whether to ensure unique
 9870        values in the output or not. If set to `True`, the function will make sure that the output values
 9871        are unique, defaults to True
 9872        :type uniquify: bool (optional)
 9873        :param annotation_field: The `annotation_field` parameter refers to the field in the VCF file that
 9874        contains the annotation information for each variant. This field is used to extract the annotation
 9875        details for further processing in the function, defaults to ANN
 9876        :type annotation_field: str (optional)
 9877        :param annotation_id: The `annotation_id` parameter in the `annotation_format_to_table` method is
 9878        used to specify the identifier for the annotation feature. This identifier will be used as a column
 9879        name in the resulting table or view that is created based on the annotation data. It helps in
 9880        uniquely identifying each annotation entry in the, defaults to Feature_ID
 9881        :type annotation_id: str (optional)
 9882        :param view_name: The `view_name` parameter in the `annotation_format_to_table` method is used to
 9883        specify the name of the temporary table that will be created to store the transformed annotation
 9884        data. This table will hold the extracted information from the annotation field in a structured
 9885        format for further processing or analysis, defaults to transcripts
 9886        :type view_name: str (optional)
 9887        :return: The function `annotation_format_to_table` is returning the name of the view created, which
 9888        is stored in the variable `view_name`.
 9889        """
 9890
 9891        # Annotation field
 9892        annotation_format = "annotation_explode"
 9893
 9894        # Transcript annotation
 9895        annotation_id = "".join(char for char in annotation_id if char.isalnum())
 9896
 9897        # Prefix
 9898        prefix = self.get_explode_infos_prefix()
 9899        if prefix:
 9900            prefix = "INFO/"
 9901
 9902        # Annotation fields
 9903        annotation_infos = prefix + annotation_field
 9904        annotation_format_infos = prefix + annotation_format
 9905
 9906        # Variants table
 9907        table_variants = self.get_table_variants()
 9908
 9909        # Header
 9910        vcf_reader = self.get_header()
 9911
 9912        # Add columns
 9913        added_columns = []
 9914
 9915        # Explode HGVS field in column
 9916        added_columns += self.explode_infos(fields=[annotation_field])
 9917
 9918        if annotation_field in vcf_reader.infos:
 9919
 9920            # Extract ANN header
 9921            ann_description = vcf_reader.infos[annotation_field].desc
 9922            pattern = r"'(.+?)'"
 9923            match = re.search(pattern, ann_description)
 9924            if match:
 9925                ann_header_match = match.group(1).split(" | ")
 9926                ann_header = []
 9927                ann_header_desc = {}
 9928                for i in range(len(ann_header_match)):
 9929                    ann_header_info = "".join(
 9930                        char for char in ann_header_match[i] if char.isalnum()
 9931                    )
 9932                    ann_header.append(ann_header_info)
 9933                    ann_header_desc[ann_header_info] = ann_header_match[i]
 9934                if not ann_header_desc:
 9935                    raise ValueError("Invalid header description format")
 9936            else:
 9937                raise ValueError("Invalid header description format")
 9938
 9939            # Create variant id
 9940            variant_id_column = self.get_variant_id_column()
 9941            added_columns += [variant_id_column]
 9942
 9943            # Create dataframe
 9944            dataframe_annotation_format = self.get_query_to_df(
 9945                f""" SELECT "#CHROM", POS, REF, ALT, "{variant_id_column}", "{annotation_infos}" FROM {table_variants} """
 9946            )
 9947
 9948            # Create annotation columns
 9949            dataframe_annotation_format[
 9950                annotation_format_infos
 9951            ] = dataframe_annotation_format[annotation_infos].apply(
 9952                lambda x: explode_annotation_format(
 9953                    annotation=str(x),
 9954                    uniquify=uniquify,
 9955                    output_format="JSON",
 9956                    prefix="",
 9957                    header=list(ann_header_desc.values()),
 9958                )
 9959            )
 9960
 9961            # Find keys
 9962            query_json = f"""SELECT distinct(unnest(json_keys({annotation_format}, '$.0'))) AS 'key' FROM dataframe_annotation_format;"""
 9963            df_keys = self.get_query_to_df(query=query_json)
 9964
 9965            # Check keys
 9966            query_json_key = []
 9967            for _, row in df_keys.iterrows():
 9968
 9969                # Key
 9970                key = row.iloc[0]
 9971
 9972                # key_clean
 9973                key_clean = "".join(char for char in key if char.isalnum())
 9974
 9975                # Type
 9976                query_json_type = f"""SELECT unnest(json_extract_string({annotation_format}, '$.*."{key}"')) AS '{key_clean}' FROM dataframe_annotation_format WHERE trim('{key}') NOT IN ('');"""
 9977
 9978                # Get DataFrame from query
 9979                df_json_type = self.get_query_to_df(query=query_json_type)
 9980
 9981                # Fill missing values with empty strings and then replace empty strings or None with NaN and drop rows with NaN
 9982                with pd.option_context("future.no_silent_downcasting", True):
 9983                    df_json_type.fillna(value="", inplace=True)
 9984                    replace_dict = {None: np.nan, "": np.nan}
 9985                    df_json_type.replace(replace_dict, inplace=True)
 9986                    df_json_type.dropna(inplace=True)
 9987
 9988                # Detect column type
 9989                column_type = detect_column_type(df_json_type[key_clean])
 9990
 9991                # Append
 9992                query_json_key.append(
 9993                    f"""NULLIF(unnest(json_extract_string({annotation_format}, '$.*."{key}"')), '')::{column_type}  AS '{prefix}{key_clean}' """
 9994                )
 9995
 9996            # Create view
 9997            query_view = f"""CREATE TEMPORARY TABLE {view_name} AS (SELECT *, {annotation_id} AS 'transcript' FROM (SELECT "#CHROM", POS, REF, ALT, {",".join(query_json_key)} FROM dataframe_annotation_format));"""
 9998            self.execute_query(query=query_view)
 9999
10000        else:
10001
10002            # Return None
10003            view_name = None
10004
10005        # Remove added columns
10006        for added_column in added_columns:
10007            self.drop_column(column=added_column)
10008
10009        return view_name

The function annotation_format_to_table converts annotation data from a VCF file into a structured table format.

Parameters

uniquify: The uniquify parameter is a boolean flag that determines whether to ensure unique values in the output or not. If set to True, the function will make sure that the output values are unique, defaults to True
annotation_field: The annotation_field parameter refers to the field in the VCF file that contains the annotation information for each variant. This field is used to extract the annotation details for further processing in the function, defaults to ANN
annotation_id: The annotation_id parameter in the annotation_format_to_table method is used to specify the identifier for the annotation feature. This identifier will be used as a column name in the resulting table or view that is created based on the annotation data. It helps in uniquely identifying each annotation entry in the, defaults to Feature_ID
view_name: The view_name parameter in the annotation_format_to_table method is used to specify the name of the temporary table that will be created to store the transformed annotation data. This table will hold the extracted information from the annotation field in a structured format for further processing or analysis, defaults to transcripts

Returns

The function annotation_format_to_table is returning the name of the view created, which is stored in the variable view_name.

def transcript_view_to_variants( self, transcripts_table: str = None, transcripts_column_id: str = None, transcripts_info_json: str = None, transcripts_info_field_json: str = None, transcripts_info_format: str = None, transcripts_info_field_format: str = None, param: dict = {}) -> bool: View Source

10011    def transcript_view_to_variants(
10012        self,
10013        transcripts_table: str = None,
10014        transcripts_column_id: str = None,
10015        transcripts_info_json: str = None,
10016        transcripts_info_field_json: str = None,
10017        transcripts_info_format: str = None,
10018        transcripts_info_field_format: str = None,
10019        param: dict = {},
10020    ) -> bool:
10021        """
10022        The `transcript_view_to_variants` function updates a variants table with information from
10023        transcripts in JSON format.
10024
10025        :param transcripts_table: The `transcripts_table` parameter is used to specify the name of the
10026        table containing the transcripts data. If this parameter is not provided, the function will
10027        attempt to retrieve it from the `param` dictionary or use a default value of "transcripts"
10028        :type transcripts_table: str
10029        :param transcripts_column_id: The `transcripts_column_id` parameter is used to specify the
10030        column in the `transcripts_table` that contains the unique identifier for each transcript. This
10031        identifier is used to match transcripts with variants in the database
10032        :type transcripts_column_id: str
10033        :param transcripts_info_json: The `transcripts_info_json` parameter is used to specify the name
10034        of the column in the variants table where the transcripts information will be stored in JSON
10035        format. This parameter allows you to define the column in the variants table that will hold the
10036        JSON-formatted information about transcripts
10037        :type transcripts_info_json: str
10038        :param transcripts_info_field_json: The `transcripts_info_field_json` parameter is used to
10039        specify the field in the VCF header that will contain information about transcripts in JSON
10040        format. This field will be added to the VCF header as an INFO field with the specified name
10041        :type transcripts_info_field_json: str
10042        :param transcripts_info_format: The `transcripts_info_format` parameter is used to specify the
10043        format of the information about transcripts that will be stored in the variants table. This
10044        format can be used to define how the transcript information will be structured or displayed
10045        within the variants table
10046        :type transcripts_info_format: str
10047        :param transcripts_info_field_format: The `transcripts_info_field_format` parameter is used to
10048        specify the field in the VCF header that will contain information about transcripts in a
10049        specific format. This field will be added to the VCF header as an INFO field with the specified
10050        name
10051        :type transcripts_info_field_format: str
10052        :param param: The `param` parameter in the `transcript_view_to_variants` method is a dictionary
10053        that contains various configuration settings related to transcripts. It is used to provide
10054        default values for certain parameters if they are not explicitly provided when calling the
10055        method. The `param` dictionary can be passed as an argument
10056        :type param: dict
10057        :return: The function `transcript_view_to_variants` returns a boolean value. It returns `True`
10058        if the operation is successful and `False` if certain conditions are not met.
10059        """
10060
10061        msg_info_prefix = "Start transcripts view to variants annotations"
10062
10063        log.debug(f"{msg_info_prefix}...")
10064
10065        # Default
10066        transcripts_table_default = "transcripts"
10067        transcripts_column_id_default = "transcript"
10068        transcripts_info_json_default = None
10069        transcripts_info_format_default = None
10070        transcripts_info_field_json_default = None
10071        transcripts_info_field_format_default = None
10072
10073        # Param
10074        if not param:
10075            param = self.get_param()
10076
10077        # Transcripts table
10078        if transcripts_table is None:
10079            transcripts_table = param.get("transcripts", {}).get(
10080                "table", transcripts_table_default
10081            )
10082
10083        # Transcripts column ID
10084        if transcripts_column_id is None:
10085            transcripts_column_id = param.get("transcripts", {}).get(
10086                "column_id", transcripts_column_id_default
10087            )
10088
10089        # Transcripts info json
10090        if transcripts_info_json is None:
10091            transcripts_info_json = param.get("transcripts", {}).get(
10092                "transcripts_info_json", transcripts_info_json_default
10093            )
10094
10095        # Transcripts info field JSON
10096        if transcripts_info_field_json is None:
10097            transcripts_info_field_json = param.get("transcripts", {}).get(
10098                "transcripts_info_field_json", transcripts_info_field_json_default
10099            )
10100        # if transcripts_info_field_json is not None and transcripts_info_json is None:
10101        #     transcripts_info_json = transcripts_info_field_json
10102
10103        # Transcripts info format
10104        if transcripts_info_format is None:
10105            transcripts_info_format = param.get("transcripts", {}).get(
10106                "transcripts_info_format", transcripts_info_format_default
10107            )
10108
10109        # Transcripts info field FORMAT
10110        if transcripts_info_field_format is None:
10111            transcripts_info_field_format = param.get("transcripts", {}).get(
10112                "transcripts_info_field_format", transcripts_info_field_format_default
10113            )
10114        # if (
10115        #     transcripts_info_field_format is not None
10116        #     and transcripts_info_format is None
10117        # ):
10118        #     transcripts_info_format = transcripts_info_field_format
10119
10120        # Variants table
10121        table_variants = self.get_table_variants()
10122
10123        # Check info columns param
10124        if (
10125            transcripts_info_json is None
10126            and transcripts_info_field_json is None
10127            and transcripts_info_format is None
10128            and transcripts_info_field_format is None
10129        ):
10130            return False
10131
10132        # Transcripts infos columns
10133        query_transcripts_infos_columns = f"""
10134            SELECT *
10135            FROM (
10136                DESCRIBE SELECT * FROM {transcripts_table}
10137                )
10138            WHERE "column_name" NOT IN ('#CHROM', 'POS', 'REF', 'ALT', '{transcripts_column_id}')
10139        """
10140        transcripts_infos_columns = list(
10141            self.get_query_to_df(query=query_transcripts_infos_columns)["column_name"]
10142        )
10143
10144        # View results
10145        clause_select = []
10146        clause_to_json = []
10147        clause_to_format = []
10148        for field in transcripts_infos_columns:
10149            clause_select.append(
10150                f""" regexp_split_to_table("{field}", ',') AS '{field}' """
10151            )
10152            clause_to_json.append(f""" '{field}': "{field}" """)
10153            clause_to_format.append(f""" "{field}" """)
10154
10155        # Update
10156        update_set_json = []
10157        update_set_format = []
10158
10159        # VCF header
10160        vcf_reader = self.get_header()
10161
10162        # Transcripts to info column in JSON
10163        if transcripts_info_json is not None:
10164
10165            # Create column on variants table
10166            self.add_column(
10167                table_name=table_variants,
10168                column_name=transcripts_info_json,
10169                column_type="JSON",
10170                default_value=None,
10171                drop=False,
10172            )
10173
10174            # Add header
10175            vcf_reader.infos[transcripts_info_json] = vcf.parser._Info(
10176                transcripts_info_json,
10177                ".",
10178                "String",
10179                "Transcripts in JSON format",
10180                "unknwon",
10181                "unknwon",
10182                self.code_type_map["String"],
10183            )
10184
10185            # Add to update
10186            update_set_json.append(
10187                f""" {transcripts_info_json}=t.{transcripts_info_json} """
10188            )
10189
10190        # Transcripts to info field in JSON
10191        if transcripts_info_field_json is not None:
10192
10193            log.debug(f"{msg_info_prefix} - Annotation in JSON format...")
10194
10195            # Add to update
10196            update_set_json.append(
10197                f""" 
10198                    INFO = concat(
10199                            CASE
10200                                WHEN INFO NOT IN ('', '.')
10201                                THEN INFO
10202                                ELSE ''
10203                            END,
10204                            CASE
10205                                WHEN CAST(t.{transcripts_info_json} AS VARCHAR) NOT IN ('', '.')
10206                                THEN concat(
10207                                    ';{transcripts_info_field_json}=',
10208                                    t.{transcripts_info_json}
10209                                )
10210                                ELSE ''
10211                            END
10212                            )
10213                """
10214            )
10215
10216            # Add header
10217            vcf_reader.infos[transcripts_info_field_json] = vcf.parser._Info(
10218                transcripts_info_field_json,
10219                ".",
10220                "String",
10221                "Transcripts in JSON format",
10222                "unknwon",
10223                "unknwon",
10224                self.code_type_map["String"],
10225            )
10226
10227        if update_set_json:
10228
10229            # Update query
10230            query_update = f"""
10231                UPDATE {table_variants}
10232                    SET {", ".join(update_set_json)}
10233                FROM
10234                (
10235                    SELECT
10236                        "#CHROM", POS, REF, ALT,
10237                            concat(
10238                            '{{',
10239                            string_agg(
10240                                '"' || "{transcripts_column_id}" || '":' ||
10241                                to_json(json_output)
10242                            ),
10243                            '}}'
10244                            )::JSON AS {transcripts_info_json}
10245                    FROM
10246                        (
10247                        SELECT
10248                            "#CHROM", POS, REF, ALT,
10249                            "{transcripts_column_id}",
10250                            to_json(
10251                                {{{",".join(clause_to_json)}}}
10252                            )::JSON AS json_output
10253                        FROM
10254                            (SELECT "#CHROM", POS, REF, ALT, "{transcripts_column_id}", {", ".join(clause_select)} FROM {transcripts_table})
10255                        WHERE "{transcripts_column_id}" IS NOT NULL
10256                        )
10257                    GROUP BY "#CHROM", POS, REF, ALT
10258                ) AS t
10259                WHERE {table_variants}."#CHROM" = t."#CHROM"
10260                    AND {table_variants}."POS" = t."POS"
10261                    AND {table_variants}."REF" = t."REF"
10262                    AND {table_variants}."ALT" = t."ALT"
10263            """
10264
10265            self.execute_query(query=query_update)
10266
10267        # Transcripts to info column in FORMAT
10268        if transcripts_info_format is not None:
10269
10270            # Create column on variants table
10271            self.add_column(
10272                table_name=table_variants,
10273                column_name=transcripts_info_format,
10274                column_type="VARCHAR",
10275                default_value=None,
10276                drop=False,
10277            )
10278
10279            # Add header
10280            vcf_reader.infos[transcripts_info_format] = vcf.parser._Info(
10281                transcripts_info_format,
10282                ".",
10283                "String",
10284                f"Transcripts annotations: 'transcript | {' | '.join(transcripts_infos_columns)}'",
10285                "unknwon",
10286                "unknwon",
10287                self.code_type_map["String"],
10288            )
10289
10290            # Add to update
10291            update_set_format.append(
10292                f""" {transcripts_info_format}=t.{transcripts_info_format} """
10293            )
10294
10295        # Transcripts to info field in JSON
10296        if transcripts_info_field_format is not None:
10297
10298            log.debug(f"{msg_info_prefix} - Annotation in structured format...")
10299
10300            # Add to update
10301            update_set_format.append(
10302                f""" 
10303                    INFO = concat(
10304                            CASE
10305                                WHEN INFO NOT IN ('', '.')
10306                                THEN INFO
10307                                ELSE ''
10308                            END,
10309                            CASE
10310                                WHEN CAST(t.{transcripts_info_format} AS VARCHAR) NOT IN ('', '.')
10311                                THEN concat(
10312                                    ';{transcripts_info_field_format}=',
10313                                    t.{transcripts_info_format}
10314                                )
10315                                ELSE ''
10316                            END
10317                            )
10318                """
10319            )
10320
10321            # Add header
10322            vcf_reader.infos[transcripts_info_field_format] = vcf.parser._Info(
10323                transcripts_info_field_format,
10324                ".",
10325                "String",
10326                f"Transcripts annotations: 'transcript | {' | '.join(transcripts_infos_columns)}'",
10327                "unknwon",
10328                "unknwon",
10329                self.code_type_map["String"],
10330            )
10331
10332        if update_set_format:
10333
10334            # Update query
10335            query_update = f"""
10336                UPDATE {table_variants}
10337                    SET {", ".join(update_set_format)}
10338                FROM
10339                (
10340                    SELECT
10341                        "#CHROM", POS, REF, ALT,
10342                            string_agg({transcripts_info_format}) AS {transcripts_info_format}
10343                    FROM 
10344                        (
10345                        SELECT
10346                            "#CHROM", POS, REF, ALT,
10347                            "{transcripts_column_id}",
10348                            concat(
10349                                "{transcripts_column_id}",
10350                                '|',
10351                                {", '|', ".join(clause_to_format)}
10352                            ) AS {transcripts_info_format}
10353                        FROM
10354                            (SELECT "#CHROM", POS, REF, ALT, "{transcripts_column_id}", {", ".join(clause_select)} FROM {transcripts_table})
10355                        )
10356                    GROUP BY "#CHROM", POS, REF, ALT
10357                ) AS t
10358                WHERE {table_variants}."#CHROM" = t."#CHROM"
10359                    AND {table_variants}."POS" = t."POS"
10360                    AND {table_variants}."REF" = t."REF"
10361                    AND {table_variants}."ALT" = t."ALT"
10362            """
10363
10364            self.execute_query(query=query_update)
10365
10366        return True

The transcript_view_to_variants function updates a variants table with information from transcripts in JSON format.

Parameters

transcripts_table: The transcripts_table parameter is used to specify the name of the table containing the transcripts data. If this parameter is not provided, the function will attempt to retrieve it from the param dictionary or use a default value of "transcripts"
transcripts_column_id: The transcripts_column_id parameter is used to specify the column in the transcripts_table that contains the unique identifier for each transcript. This identifier is used to match transcripts with variants in the database
transcripts_info_json: The transcripts_info_json parameter is used to specify the name of the column in the variants table where the transcripts information will be stored in JSON format. This parameter allows you to define the column in the variants table that will hold the JSON-formatted information about transcripts
transcripts_info_field_json: The transcripts_info_field_json parameter is used to specify the field in the VCF header that will contain information about transcripts in JSON format. This field will be added to the VCF header as an INFO field with the specified name
transcripts_info_format: The transcripts_info_format parameter is used to specify the format of the information about transcripts that will be stored in the variants table. This format can be used to define how the transcript information will be structured or displayed within the variants table
transcripts_info_field_format: The transcripts_info_field_format parameter is used to specify the field in the VCF header that will contain information about transcripts in a specific format. This field will be added to the VCF header as an INFO field with the specified name
param: The param parameter in the transcript_view_to_variants method is a dictionary that contains various configuration settings related to transcripts. It is used to provide default values for certain parameters if they are not explicitly provided when calling the method. The param dictionary can be passed as an argument

Returns

The function transcript_view_to_variants returns a boolean value. It returns True if the operation is successful and False if certain conditions are not met.